Apache Arrow Flight: gRPC Streaming for Parquet on S3 (Python Demo)

Arrow Flight is more than “just a file format.” In this quick walkthrough, I break down what Flight is, why gRPC makes it perfect for streaming Arrow record batches, and how the core RPC methods work (ListFlights + DoGet). Then we build a tiny Python Flight server that: - Lists Parquet files in a public S3 bucket - Uses Flight methods to return schemas and stream data back to a local client If you want a fast mental model for Flight, plus a practical “hello world” you can adapt for real services, this is it. 🔗 Useful Links: - Github Code: https://github.com/early-signal-tech/arrow-flight-examples/blob/main/arrow-flight-simple.py - Arrow Flight docs: https://arrow.apache.org/docs/format/Flight.html - My Substack for long-form data engineering content: https://thefulldatastack.substack.com/ - Connect on LinkedIn: https://www.linkedin.com/in/hoytemerson/ - Learn about my technical content company, Early Signal: https://www.earlysignal.tech/ If you found this helpful, like and subscribe for more Arrow, DuckDB, and lakehouse quick wins.