From Product Need to Distributed Vector Search: Lessons from a Criteo Use Case | Milvus Meetup

Criteo picks which product to show in an ad from a catalogue of more than 12 billion products, and has about 25 ms to do it. Peter Goron (Senior Staff SRE Lead) explains why Criteo’s in-house FAISS-based KNN service stopped scaling as new AI use cases arrived. He covers how the team benchmarked OpenSearch, Aerospike and Milvus, and why Milvus won: it’s distributed, open source, runs on-premise and scales each component on its own. He then describes the step-by-step rollout, which now reaches 5 ms p99 on product recommendation. Mehdi Sebbar (Staff MLE) then walks through Criteo’s agentic product recommendation service, which turns a conversation into 5–10 relevant products. An LLM extracts queries and filters from the conversation, and the service runs 3–5 parallel Milvus searches over a custom Matryoshka embedding model. An LLM then checks the candidates for relevance before the final ranking. He also covers the daily catalogue pipeline (Spark ingestion with alias swaps) and how the team cut query latency from about 300 ms to under 20 ms. What you’ll learn: Why a FAISS library service hits a wall once many teams share it, and what a distributed vector database changes How to evaluate vector databases against on-prem, open-source and latency requirements An LLM + vector search pipeline for conversational product recommendation Using filtering, distance thresholds (radius), grouping and parallel searches in a single query Daily rebuilds of a catalogue with billions of products, using delta ingestion, a collection per region and alias switching Latency tuning: drop strong consistency when you don’t need it, and set a radius instead of shrinking top-K Chapters 0:00 Introduction 1:24 Real-time ad recommendation at Criteo 4:35 The in-house FAISS KNN service 5:40 Why Criteo needed a platform service 6:17 Requirements and benchmark: OpenSearch vs Aerospike vs Milvus 8:49 Milvus architecture and independent scaling 10:32 Progressive adoption and results 13:10 Agentic product recommendation 15:04 Conversational recommendation demo 18:53 Where Milvus sits in the pipeline 22:41 From catalogue to collection, every day 24:58 Milvus features in use: filters, radius, grouping, parallel search 26:38 Learning on the job: latency tuning 28:40 Takeaways 29:49 Q&A This talk was recorded at the Milvus Meetup Paris, hosted by Criteo. The event brought together vector search practitioners to explore the algorithms, infrastructure and production patterns behind modern AI-powered search. Find us on: Zilliz Website: https://zilliz.com Milvus Website: https://milvus.io Zilliz LinkedIn: https://www.linkedin.com/company/zilliz/ Milvus LinkedIn: https://linkedin.com/company/67143129 Milvus X: https://x.com/milvusio Milvus GitHub: https://github.com/milvus-io/milvus Try Zilliz Cloud: https://cloud.zilliz.com/signup