Engineering Dynamic Lineage: Column-level lineage using Open Lineage, Spark, and Delta Lake

Traditional static lineage tools fail to track real-time data flow in complex enterprise environments. This session explores dynamic, deterministic lineage as telemetry—treating query execution as parseable events that reveal current data state. We'll examine the Open Lineage schema as an open-source alternative to proprietary vendor solutions, demonstrating how to stitch together data flows across thousands of jobs. Through live demos and architectural discussion, we'll show how lineage integrates with the lakehouse stack alongside tables — providing a natural view of your upstream and downstream data dependencies. Learn practical approaches for implementing lineage using Spark listeners and open standards that work across platforms. Speaker: Raki Rahman, Microsoft Host: Scott Haines, Databricks