How we built a DISTRIBUTED search engine in Rust

In a data-driven world, having to search through terabytes of logs is common. This simple task can turn out to be daunting and very expensive. This talk will present how to implement such an engine with an efficient architecture strongly inspired by Snowflake which separated compute and storage. 🦀 Thank you RustLab for organizing the event! Check out more of their events on their Youtube channel: @rustlabconference3671 🔎 Quickwit Repo: https://github.com/quickwit-oss/quickwit 🏇 Tantivy Repo: https://github.com/quickwit-oss/tantivy 🚀Quickstart with Quickwit: https://quickwit.io/docs/get-started/quickstart _____ 🕰️ TIMECODE : 0:00 Introduction & agenda 2:45 Search architecture 3:50 Tantivy indexing & querying 5:54 Data structures 10:30 Incremental indexing 12:53 Shared-nothing architecture 17:15 Importance of log search 20:25 Shared-disk architecture 29:09 Problem 1: Staying truly stateless 31:58 Problem 2: Searching with a bad throughput 33:07 Problem 3: Searching in a high latency world 33:59 Demo time 42:50 What we learnt after 2 years 45:40 Choosing an actor framework 51:51 Questions _____ ⭐️ Psst...on GitHub there is a Star button...you know...only if you like our content : https://github.com/quickwit-oss/quickwit 🖤 Subscribe for more videos about software engineering & architecture : https://www.youtube.com/channel/UCvZV... 💬 Join our community : - Mind-blowing Discord : https://discord.gg/q5fquGeR3D - Sky-breaking blogs : https://quickwit.io/blog - Latest & greatest news on Twitter : https://twitter.com/Quickwit_Inc - Cosmos-litting videos : @quickwit8103 Topics: indexing, search, data, query, architecture, distributed queues, messaging queues, metastore, metadata, indexer, scaling indexes, apache kafka, search engine internals, log management & analytics, shards #softwareengineer #software #rust