Vector Search on Apache Iceberg & Delta Lake — Without Copying Data

How can you run vector search directly on Apache Iceberg or Delta Lake without copying embeddings into a separate vector database? In this technical session, we explore how Apache Pinot enables vector similarity search directly on data stored in Apache Iceberg and Delta Lake. Instead of maintaining duplicate copies of embeddings and synchronization pipelines between your lakehouse and a vector database, this architecture keeps the data lake as the source of truth while adding low-latency vector retrieval. You’ll learn how approximate nearest neighbor (ANN) search and HNSW vector indexes can be adapted for lakehouse data stored in object storage, including Amazon S3. The session also covers how Apache Pinot External Tables make it possible to query lakehouse data and combine semantic vector search with SQL filters over the same dataset. Topics covered: * How to run vector search directly on Apache Iceberg and Delta Lake * Using Apache Pinot for vector similarity search over lakehouse data * How HNSW indexes support approximate nearest neighbor (ANN) search * Adapting vector indexes for object storage such as Amazon S3 * Using Apache Pinot External Tables with Iceberg and Delta Lake * Combining semantic search with SQL filters * Reducing data duplication between lakehouse and vector database systems * Architectural tradeoffs of adding vector search directly to the data lake If your embeddings already live in Apache Iceberg or Delta Lake, this session walks through an alternative to moving that data into another purpose-built vector database—and the architecture required to make vector retrieval practical directly from the lakehouse. LEARN MORE ► Learn about StarTree and Apache Pinot™: https://docs.startree.ai/ CONNECT Subscribe on YouTube: https://stree.ai/sub Community Slack: https://stree.ai/join-slack Twitter: https://twitter.com/startreedata LinkedIn: https://www.linkedin.com/company/startreedata GitHub: https://github.com/startreedata Site: https://startree.ai ABOUT STARTREE StarTree, powered by Apache Pinot™, is the platform for continuous insights at scale. StarTree turns raw data streams—like clickstreams, logs, vector embeddings, and sensor data—into real-time analytics. It powers customer-facing data products, agent-facing data products, and operational intelligence. Apache Pinot™ delivers real-time analytics by design, rather than extension—built from the ground up to ingest massive data streams with millisecond freshness, execute sub-second aggregations, and support tens of thousands of queries per second. Pinot achieves this while radically reducing infrastructure costs by scanning less data and extreme compression. StarTree was founded by the core software engineering team and inventors of Apache Pinot™, which currently powers hundreds of user-facing applications at companies across industries, including Uber, LinkedIn, Stripe, Target, 7Eleven, NuBank, Etsy, Walmart, Together AI, and more. #ApachePinot #VectorSearch #ApacheIceberg #DeltaLake #lakehouse