'Beam'-ing Iceberg Tables Across Environments Using Project Nessie - Beam Summit 2026

Presented by Vineel Arekapudi at Beam Summit 2026. At Wells Fargo, we delve into our approach for backing up and synchronizing Apache Iceberg tables across environments using Project Nessie as a catalog-level control plane and Apache Beam as the unified replication engine. By combining object storage replication with Nessie's Git-like metadata versioning, orchestrated through a single Beam pipeline. We demonstrate how production Iceberg tables can be continuously mirrored into non-production catalogs without low-level database syncs. The architecture consists of two coordinated replication layers, implemented as a unified Apache Beam pipeline: 1. Storage-Layer Replication A Beam pipeline reads all Iceberg table data that's in Parquet file format along with Metadata files like manifests, and metadata JSON directly from production S3 and writes to non-production S3. Beam's parallel execution model handles fan-out across thousands of tables efficiently, replacing ad-hoc tooling like rclone or distcp with a testable, portable, and observable pipeline. 2. Catalog-Layer Replication with Nessie, Powered by Beam, Not Raw MongoDB Sync Rather than directly synchronizing Nessie's underlying MongoDB collections (objs2, refs2), an approach that is fragile, version-sensitive, and opaque, we use Apache Beam as an abstraction layer that operates at the Nessie API level. This means replication is driven by Nessie's own versioning semantics rather than internal MongoDB implementation details making the approach more resilient to Nessie upgrades. A Unified Pipeline with Batch and Streaming Modes Both layers run within the same Beam pipeline, giving us a single model for two distinct operational needs. In streaming mode, the pipeline continuously watches for new Nessie commits and triggers incremental storage and catalog replication, keeping non-production environments near-current with production. In batch mode, the same pipeline handles full environment bootstrapping or point-in-time recovery to a specific Nessie snapshot. Beam's runner portability was essential in our regulated environment: pipelines are developed and validated locally using the Direct Runner before being deployed to our Spark cluster via the Spark Runner, without any rewrite. Once completed, the non-production Nessie catalog becomes a true, API-level mirror of production. We will share practical lessons learned, including Nessie API pagination at scale, handling Beam pipeline failures mid-replication, and ensuring catalog consistency when storage and metadata sync are not atomic.