ChitChat: Cluster Membership with Failure Detection in Rust | Evance, Software Engineer

Cluster membership management is one of the critical aspects of a distributed system. To put it simply, nothing can work if nodes don’t know about each other and what service or resource each can offer. In this talk, first, we briefly describe Quickwit’s previous cluster management implementation. Then deeply explore the new implementation based on the scuttlebutt algorithm while highlighting our motivations and what the current rust ecosystem offers. The algorithm presented here is also what powers Apache Cassandra cluster feature. We finally conclude by presenting a couple of tricks we used to overcome some implementation challenges while contrasting with the solutions adopted by Apache Cassandra. ⚙️Thank you Rust London for organizing the event! Check out more of their events on their Youtube channel: https://www.youtube.com/channel/UCJKVYbQ2iJkxPD0Hax-ohXg 🦀 ChitChat Repo: https://github.com/quickwit-oss/chitchat _____ 🕰️ TIMECODE : 0:00 What is Quickwit? 1:27 Introduction 1:58 What is Cluster Membership Management 3:17 ChitChat as a Rust Library 4:08 Moving Away from S.W.I.M 6:14 Scuttlebutt Algorithm 9:14 How Scuttlebutt works? 10:50 How Scuttlebutt gossips? 11:46 What a Scuttlebutt gossip round is made of? 13:30 What about Faulty/Slow Nodes? 15:05 Phi Accrual Detection Algorithm 16:45 Integrating ChitChat within Quickwit 18:20 Our Solution - Node Identification 21:47 Q&A Section 31:15 Outro _____ ⭐️ Psst...on GitHub there is a Star button...you know...only if you like our content : https://github.com/quickwit-oss/quickwit 🖤 Subscribe for more videos about software engineering & architecture : https://www.youtube.com/channel/UCvZVuRm2FiDq1_ul0mY85wA ✏️ References: ScuttleButt paper: https://www.cs.cornell.edu/home/rvr/papers/flowgossip.pdf Phi Accrual error detection: https://www.researchgate.net/publication/29682135_The_ph_accrual_failure_detector Cassandra details: https://www.youtube.com/watch?v=FuP1Fvrv6ZQ https://docs.datastax.com/en/articles/cassandra/cassandrathenandnow.html https://github.com/apache/cassandra/blob/f5fb1b0bd32b5dc7da13ec66d43acbdad7fe9dbf/src/java/org/apache/cassandra/gms/Gossiper.java#L1749 💬 Join our community : - Mind-blowing Discord : https://discord.gg/q5fquGeR3D - Sky-breaking blogs : https://quickwit.io/blog - Lightning-speed Quickwit installation : https://quickwit.io/docs/get-started/quickstart Latest & greatest news on Twitter : https://twitter.com/Quickwit_Inc - Rust-mastery Twitch : https://www.twitch.tv/quickwit_io - Cosmos-litting videos : https://www.youtube.com/channel/UCvZVuRm2FiDq1_ul0mY85wA Topics: indexing, search, data, query, architecture, sre, devops, metastore, metadata, indexer, scaling indexes, cluster membership, search engine internals, devsecops, shards #softwareengineer #rust #devops