Building a Highly Efficient Search Engine for Logs In Rust | Francois, the Co-Founder of Quickwit
In today’s modern world, any systems/servers/applications produce an increasing tremendous amount of logs and you often have to choose between costly SaaS services or on-premise solutions that are hard to manage at a large scale. That’s why we build Quickwit from the ground up to make it cloud-native. In this talk, we will see how we re-designed indexing and search to truly decouple storage and compute, and create stateless search instances while keeping a sub-second response time. ⚙️Thank you Rust London for organizing the event! Check out more of their events on their Youtube channel: https://www.youtube.com/channel/UCJKVYbQ2iJkxPD0Hax-ohXg 🔎 Quickwit Repo: https://github.com/quickwit-oss/quickwit 🏇 Tantivy Repo: https://github.com/quickwit-oss/tantivy 🚀Quickstart with Quickwit: https://quickwit.io/docs/get-started/quickstart _____ 🕰️ TIMECODE : 0:00 Introduction 0:39 Speaker Bio 1:39 Overview of a Search Engine 3:35 Log Search Characteristics 6:44 Shared Nothing Architecture 13:35 Pros & Cons of Shared Nothing Architecture 15:15 Shared Disk Architecture 19:33 3 Problems to Solve 20:22 Problem 1: Staying truly stateless 22:31 Problem 2: Search TBs with a bad throughput 24:41 Problem 3: Search in a High-Latency World 27:30 Q&A Section 35:30 Outro _____ ⭐️ Psst...on GitHub there is a Star button...you know...only if you like our content : https://github.com/quickwit-oss/quickwit 🖤 Subscribe for more videos about software engineering & architecture : https://www.youtube.com/channel/UCvZVuRm2FiDq1_ul0mY85wA 💬 Join our community : - Mind-blowing Discord : https://discord.gg/q5fquGeR3D - Sky-breaking blogs : https://quickwit.io/blog - Latest & greatest news on Twitter : https://twitter.com/Quickwit_Inc - Rust-mastery Twitch : https://www.twitch.tv/quickwit_io - Cosmos-litting videos : https://www.youtube.com/channel/UCvZV... Topics: indexing, search, data, query, architecture, distributed queues, messaging queues, metastore, metadata, indexer, scaling indexes, apache kafka, search engine internals, log management & analytics, shards #softwareengineer #software #rust #developer #programming