Cost-Efficient Rust in Practice, Rust Tokyo Meetup | Paul Masurel, Co-Founder
How can Tantivy index the entire English Wikipedia on a single desktop in just a few minutes? How does Quickwit reach a throughput of 1GB/s on a k8s cluster? Paul Masurel (Co-Founder & CEO of Quickwit) will tell about building an open-source, cloud-native search engine from the ground up in Rust. Watch to learn about the features, design, internals, and challenges of developing Tantivy and Quickwit. 🦀Thank you Rust Tokyo for organizing the event! Check out their content: - Their Youtube Channel: https://www.youtube.com/channel/UCK0U2zrJDzqe8WwFYQSck_g - Their Meetup: https://www.meetup.com/tokyo-rust-meetup/ 🕰️ TIMECODE : 0:00 Meetup Introduction 1:20 Speaker Bio 4:20 Tantivy Search Benchmark 6:40 Plan of Presentation 7:08 Tantivy's Index & Search Side 10:35 What Makes Tantivy Fast? 14:08 Inverted Index 15:42 Data Layout 21:04 Skip Index, Block of Documents, Document Seeking 25:06 How to Seek into the Right Block of Documents 29:54 Rust is low-level optimization friendly 31:30 Different DocSet traits used & DocSet Intersection 34:30 What is so Special about Quickwit's Log Search? 37:45 What makes Quickwit so cost-efficient? 40:55 Shared-Nothing Architecture Pros & Cons 43:31 Network is faster: Time for Shared Disk Architecture 45:58 Healthy System Load 49:00 Q&A Time ___ ⭐️ Psst...on GitHub there is a Star button...you know...only if you like our content : https://github.com/quickwit-oss/quickwit 🏇 Tantivy Repo: https://github.com/quickwit-oss/tantivy 🖤 Subscribe for more videos about software engineering & architecture : https://www.youtube.com/channel/UCvZV... 💬 Join our community : - Mind-blowing Discord : https://discord.gg/q5fquGeR3D - Sky-breaking blogs : https://quickwit.io/blog - Lightning-speed Quickwit installation : https://quickwit.io/docs/ - Latest & greatest news on Twitter : https://twitter.com/Quickwit_Inc - Cosmos-litting videos: https://www.youtube.com/channel/UCvZVuRm2FiDq1_ul0mY85wA Topics: indexing, search, data, query, architecture, sre, devops, metastore, metadata, indexer, scaling indexes, cluster membership, search engine internals, devsecops, shards, search engine library, architecture, software architecture #softwareengineer #rustlang #devops