Latest AI and tech news

When I first read Kafka creator Jay Kreps’ article about logs (and his subsequent book, I ♥ Logs), I was fascinated by the elegance of immutable, append-only sequences of records....

Since AWS announced S3 Tables last December, many industry experts have shared their opinions through blogs and social media. Most of the feedback I've read so far has been somewhat negative. Critics like Daniel argue that S3 Tables create vendor loc...

The open source data engineering landscape continues to evolve rapidly, with significant developments across storage, processing, integration, and analytics in 2024....

This is the fifth part in the Data Landscape Trends 2024-2025 series, focusing on the state of the open source workflow orchestration systems....

This is the forth part in the Data Landscape Trends 2024-2025 series, focusing on the state of the open-source real-time OLAP database systems....

This is the third part in the Data Landscape Trends 2024-2025 series, focusing on the evolution of zero-disk architecture....

This is part two of Data Landscape Trends 2024-2025 series, focusing on single-node processing trends....

As we dive into 2025, the data engineering field continues its dramatic evolution. In this series, we'll explore the transformative trends reshaping the data engineering landscape, from emerging architectural patterns to new tooling approaches....

In a recent blog post, Jordan Tigani, MotherDuck co-founder and former tech lead at Google BigQuery, highlighted that most companies don't actually deal with "Big Data." An analysis of half a billion sample queries run on Amazon Redshift revealed th...

DuckDB, a high-performance, embeddable analytical engine, has been generating a lot of interest due to its lightweight setup and powerful capabilities....

In July this year, Snowflake open-sourced its Polaris Catalog under the Apache 2.0 license, with plans to submit it to the Apache Incubator program. Polaris is a catalog service designed for Apache Iceberg but can extend to other major open formats a...

After years of working in the data space, you witness the rise and fall of numerous tools and products. Over time, you become more skeptical of the latest shiny tools, often dismissing them as marketing-driven hype, until you reach a point where they...

Modern Open Table Formats, such as Hudi, Delta Lake, and Iceberg, are built on the foundation of a file-based metadata layer. In these formats, all metadata related to tables, columns, data files, and partitions is stored in metadata files alongside ...

The crowded field of open-source engines has just welcomed a new player: Maestro, recently open-sourced by Netflix!...

Welcome to the Data Engineering Digest (DLD) newsletter series. This will be a periodic roundup of the latest and greatest in the world of data & data engineering in particular. We'll deliver a curated selection of finest news articles, blog posts, t...

In Part I we went over the origin and architecture of traditional table management systems and the first generation of Open Table Formats (OTF). In this final part, I will discuss the second and third generation OTFs....

If you have been following trends in data engineering landscape over the past few years surely you have been hearing a lot about Open Table Formats and Data Lakehouse, if not already working with them! But what is all the hype about table formats if ...

While the widespread hype surrounding Generative AI and ChatGPT took the tech world by storm, 2023 witnessed yet another exciting and vibrant year in the data engineering landscape, steadily grown more diverse and sophisticated, with continuous innov...

In my previous blog post I covered different relational database data extraction techniques when using batch or micro-batch data collection frameworks....

Extracting data from relational databases such as those used in Business Support Systems (BSS) is a common ETL and data extraction pattern in analytical data platforms. It’s the job of data engineers to choose the best approach to build RDBMS data in...