When I first read Kafka creator Jay Kreps’ article about logs (and his subsequent book, I ♥ Logs), I was fascinated by the elegance of immutable, append-only sequences of records.

Logs represent one of the simplest yet most powerful abstractions for managing data on physical storage. Since the invention of Apache Kafka and similar log-structured systems, the big data community has embraced the simplicity and scalability of log-oriented semantics.

This paradigm shift manifested most notably in modern data lakehouse table formats like Apache Iceberg, Apache Hudi and Delta Lake where—unlike traditional database implementations—all metadata operations are captured in immutable logs stored alongside data files on cloud storage such as S3.

But then comes DuckLake, a new open table format introduced by the DuckDB creators who have boldly rejected the log-oriented metadata philosophy.

These contrarians seem determined to challenge big data and distributed systems conventions built over the last decade. First, they built DuckDB to argue you don’t need distributed systems like Spark for majority of workloads. Now they’ve built DuckLake to claim you don’t need distributed metadata logs and Iceberg!

This article is going to examine DuckLake’s metadata design in comparison to current and previous generation table formats.

To do an architectural comparison, first let’s look at the primary design goals of modern log-oriented table formats.

Log-Oriented Open Table Format Design

Modern open table formats store metadata in immutable files right alongside the data for each dataset in object storage. Why is that advantageous?

This enables independent, horizontal scaling of both data and metadata. There’s no need to manage external metadata servers, no backend Metastore to become a choke point, and a single protocol (storage REST API) to handle everything, both data and metadata.

With this scalable architectural foundation, open table formats such as Apache Iceberg, Apache Hudi, and Delta Lake have emerged as the dominant approach for managing data on cloud object stores.

Now, with DuckLake entering the scene, an obvious question arises: why build yet another lakehouse table format when the market already seems saturated? What gap could DuckLake possibly fill, and what makes it different from existing table format architectures?

To answer that, let’s first briefly recap how we got here and why major open table formats adopted their current log-oriented metadata architecture.

Understanding this history is essential because DuckLake’s design bears similarities to the previous generation Apache Hive system.

The Hadoop & Apache Hive Era

In the Hadoop era, Apache Hive (+ Hive Metastore) served as the dominant table format on data lakes.

However, Hive suffered from two critical bottlenecks that motivated top tech companies like Uber and Netflix to develop new table formats for managing petabyte-scale data lake platforms:

1. Slow Metadata Operations During Query Planning

Hive table format was a directory-oriented format that relied heavily on directory and file listing operations during the query planning phase.

The critical problem for Netflix and similar companies managing their big data platforms on cloud, was that Hive table format was architected for Hadoop HDFS, not cloud object storage like S3.

When migrating from Hadoop HDFS storage to cloud object storage like S3, the I/O patterns change dramatically: fast local RPC calls are replaced with slower REST API calls. Additionally, cloud storage introduces new constraints such as Per-API-call charges, API limits (e.g., 1,000 objects per list operation) and rate limits that didn’t exist in Hadoop/HDFS environments.

For tables with millions of data files across many partitions, query planning becomes both painfully slow and expensive.

Listing all files, issuing HEAD requests, and reading metadata sections from millions of Parquet files—each with network round-trip latencies up to 100ms—creates a significant performance bottleneck.

2. Inefficient Transaction and Mutable Data Management

Beyond performance issues, Hive lacked efficient transaction and concurrency management, and relied on pessimistic locking mechanisms often resulting in long lock contentions.

To make matters worse, other engines like Trino didn’t respect Hive’s locking mechanisms and bypassed them entirely, creating data consistency risks.

For a deeper dive into this history, I’ve previously written a comprehensive two-part series on the evolution of open table formats:

Emergence of New Open Table Formats

Facing these limitations, tech giants like Netflix and Uber recognised the need for a fundamentally different architecture. A new table format that can handle massive workloads far more efficiently.

Two key design requirements emerged to improve query planning performance:

1. Store the list of all data files in the metadata layer to avoid excessive and slow object listing calls to storage APIs during query planning.

2. Store data file column statistics (such as min/max values) in the metadata layer to eliminate the need to read and parse built-in column stats from each Parquet or ORC data file at query execution time, enabling efficient file-level data file pruning (often referred to as predicate pushdown) right at the query planning time.

But this raised a critical question: where should these massive amounts of file and column statistics be stored?

For a typical petabyte-scale data lake with hundreds of millions of data files—and tables averaging around 10 columns—you’re looking at billions of metadata records for column statistics alone.

I imagine the engineers designing these table formats took a hard look at Hive Metastore’s architecture and quickly rejected the idea of storing such massive metadata volumes in a centralised database system. Even with continuous scale-up efforts, Hive Metastore was already becoming a bottleneck—adding billions of metadata records would only make matters worse.

Combine this realisation with the proven scalability of distributed logs, and storing all metadata on object storage became the obvious choice. This approach delivers a shared-nothing metadata architecture with theoretically infinite scalability.

Similar to how data files are distributed, metadata is partitioned per dataset, allowing each table to manage its own metadata independently.

With this design, a table’s metadata can grow linearly with the data itself without ever becoming a system-level bottleneck.

This design has made many database architects sceptical. After all, HTTP wasn’t designed for the low-latency, high-concurrency demands of database storage—particularly for metadata management, which requires fast reads, writes, and atomic updates.

Despite these scepticism, modern open table formats have gained widespread adoption across the data community.

This widespread adoption of log-oriented formats might seem like vindication of the architecture, but success doesn’t mean perfection. Every design involves compromises.

The Tradeoffs

No design comes without tradeoffs. The pain just shifts from one layer to another, and the distributed log-oriented metadata architecture is no exception.

But what could possibly go wrong? On paper, it all sounds great. In practice, things get complicated.

It turns out metadata has fundamentally different performance and management characteristics than data itself. While large-scale data processing optimises mainly for high throughput, metadata needs to be written, read, updated, deleted and rewritten quickly, reliably, and atomically with low latency.

By design, object stores are not the best match for such workloads.

Challenge #1: Metadata Amplification & Operational Complexity

A major issue with new-generation open table formats is excessive metadata file generation.

Since underlying storage is immutable, each DML operation (insert, update, delete) requires creating a new (or multiple) metadata log file.

In Iceberg, every write generates a new Manifest file, a new Manifest List, and a new JSON Metadata file. On platforms with heavy updates, millions of tiny log files can be generated in a short period of time.

This quickly becomes counterproductive without constant housekeeping to merge and remove obsolete log files. Ironically, the original goal of log-based metadata was to eliminate excessive storage API calls during query planning—but now the problem has simply shifted to a different layer: metadata itself.

How is this handled? Through constant housekeeping operations: expiring older snapshots, background compaction jobs continuously merging small metadata log files into larger ones and dropping obsolete files.

Challenge #2: Atomicity & Concurrency Control

While all new open table formats support read-write isolations through Snapshot Isolation and MVCC, Atomicity on the storage layer and support for multi-write concurrency control (to same object) have been challenging across different storage backends.

Most distributed cloud storage systems lack full ACID compliance and typically provide only eventual consistency guarantees. For instance, AWS S3 doesn’t support atomic file renames. Atomic put-if-absent only became available on AWS recently.

Apache Iceberg relies on an external catalog’s atomic commit (such as a transactional database) for atomically update the pointer to the latest metadata file containing the latest snapshot Id.

To support concurrent writes, Delta often uses a lightweight external lock (commonly DynamoDB-based) to coordinate exclusive access to the log files during commit to deal with race condition of two writes trying to create the same log file (with same version number).

Apache Hudi additionally supports table-level locks through external services such as Zookeeper, Hive Metastore, or DynamoDB to coordinate concurrent writers.

Challenge #3: Dependency on External Catalog Service

The third major challenge is the lack of a native fully-featured catalog. This partially explains the data catalog wars in 2024, when major vendors competed to become the dominant catalog service for data lakehouse platforms.

While schemas are stored and evolved within the metadata layer, allowing direct table interaction without a central catalog, these formats lack the infrastructure for scalable data discovery, governance, and business-level metadata management that users and external query engines require.

The following diagram shows a typical open lakehouse architecture, with an open catalog as a key component of the architecture.

Having outlined these challenges, it’s worth revisiting Hive Metastore to recognise its inherent strengths despite its weaknesses and scalability issues.

First, it was backed by transactional SQL databases such as MySQL and PostgreSQL, which made managing metadata operations in a concurrent, transaction-safe manner straightforward with full CRUD support.

Second, the Metastore served dual roles as both the metadata backend and the catalog for managing schemas. This meant users and all external query engines—Athena, Spark, Trino, Presto—could access and manage schemas directly through Hive Metastore component without requiring any other external catalog service.

These observations set the stage for understanding DuckLake’s philosophy.

Back to DuckLake

Now that we have the required background info let’s go back to DuckLake.

I believe DuckLake aims to achieve the best of both worlds: return to familiar, reliable, update-friendly, and transactionally safe database systems for managing metadata, while incorporating the query performance and table management features pioneered by modern open table formats.

DuckLake is built on a foundational premise:

The DuckLake creators observed the challenges open table formats have faced over the past five years and reached a surprising conclusion: perhaps using a centralised transactional SQL database for metadata management—as Hive did—wasn’t such a flawed approach after all. The key is to preserve what worked while fixing what didn’t.

What exactly is DuckLake’s critique of the log-oriented approach?

Managing table metadata through scattered, immutable log files on object stores creates significant operational complexity.

The challenges include handling excessive numbers of small metadata files, high latency when traversing and scanning metadata logs, and managing the complete lifecycle of table snapshots, version files, data file metadata, partition information, and column statistics.

Therefore, DuckLake proposes moving all scattered metadata structures back to a familiar, reliable centralised SQL database. But wait... haven’t we been here before? Isn’t this a step backward to Apache Hive + Metastore architecture?

Yes, it’s a small step backward—but moving backward isn’t always a bad thing. The key difference between DuckLake and Hive lies in storing complete data file details in the Metastore.

Remember, one of Hive’s major performance bottlenecks was expensive LIST calls to discover all data files for each query plus column stats gathering during query planning phase.

DuckLake eliminates this bottleneck by tracking all data files and storing column-level statistics for each file in the metadata layer, just like modern open table formats. The critical difference is that these details live in relational database tables rather than scattered log files.

Additionally, DuckLake maintains table snapshot information directly in the Metastore, simplifying snapshot history tracking and enabling snapshot isolation through MVCC.

DuckLake follows the proven approach of modern OLAP systems like BigQuery and Snowflake, which successfully manage data on object storage while maintaining all metadata in database systems.

This all sounds compelling—simpler operations, transactional guarantees, familiar SQL interfaces. But there’s an elephant in the room that we must address.

What about Scalability?

Here comes the pivotal question: can DuckLake scale?

After all, scalability challenges were also part of the reason why Iceberg and Hudi were built to replace Hive table format.

A full-featured DuckLake implementation will likely struggle to scale to very large, petabyte-scale platforms, without significant database tuning, vertical scaling and potentially using a distributed SQL database system (ex Neon). However, not everyone operates at that scale—so DuckLake will likely work well without running into metadata bottleneck issues for the majority of workloads.

The Scalability Challenge

DuckLake stores everything in a centralised managed metadata database. Certain metadata tables will accumulate substantial volume of data:

1. data_file table: Stores the list of Parquet data files for each table

2. delete_file table: Tracks deleted Parquet files for each table

3. file_column_statistics table: Stores column-level statistics (min/max, null count, etc.) for each column in each Parquet file

Consider a data lake with one petabyte of data—and we’re not even discussing companies managing tens or hundreds of petabytes. A typical 1 PB data lake contains approximately 50-100 million data files. With optimisation and reasonable file sizes, you might reduce this somewhat, but the order of magnitude remains.

Assuming an average of 10 columns per table, this generates:

• 100 million records in the data_file table

• 1 billion records in the file_column_statistics table (100M files × 10 columns)

• Hundreds of millions of records in the delete_file table over time from merge and insert-overwrite workloads

This doesn’t account for historical records from expired snapshots (before expiring snapshots and compaction), which increases volumes further.

Query-Time Implications

Storing and writing to tables with 100s of millions or billions of rows may not present a significant challenge, provided the database server has adequate storage and the tables aren’t heavily indexed in ways that slow down writes.

The real performance challenge lies in querying these tables. Consider querying historical data containing millions of files with many columns. For a typical large query scanning 1 million data files with 20 columns, the metadata stage must:

1. Select 1 million records from the data_file table

2. Join against records from the delete_file table to filter out deleted files

3. Scan 20 million records (20 column stats records per data file) from the file_column_statistics table

However, we don’t need to scan and read all these records. Remember, we’re in the SQL world—we can easily filter metadata and only retrieve the candidate list of data files that must be scanned for matching records.

Here’s a sample query to retrieve the list of data files from a clickstream table where the query filters by event_timestamp column:

SELECT data_file_id
FROM file_column_stats
WHERE
  table_id = 'clickstream' AND
  column_id = 'event_timestamp' AND
  ('timestamp-value' >= min_value OR min_value IS NULL) AND
  ('timestamp-value' <= max_value OR max_value IS NULL);

DuckLake Creators’ Perspective

In a recent presentation, Prof. Hannes Mühleisen claims that DuckLake can easily handle petabyte-scale data, citing tests conducted on a table with 100 million snapshots.

Jordan Tigani draws parallels between DuckLake’s column statistics storage and BigQuery’s CMETA metadata table, which serves a similar purpose. However, according to BigQuery’s published research paper on the subject, BigQuery stores column-level statistics separately for each individual table rather than in a single central metadata table.

This means, similar to how existing lakehouse formats manage metadata independently per dataset, there exists a one-to-one mapping between each user table (e.g., sales) and its corresponding column metadata statistics table (e.g., cmeta_sales). This design avoids possible large central metadata tables and enables horizontal scalability.

Similarly, Snowflake stores such metadata on its highly optimised distributed metadata layer, which appears to be built on the FoundationDB distributed key-value store.

As Jordan emphasises, nothing inherently prevents a DuckLake implementation from using a similar scalable and distributed storage engine for the metadata layer.

A vendor with sufficient expertise and resources—such as MotherDuck itself—could reasonably manage such infrastructure, much as Google and Snowflake do. However, for self-hosted deployments, implementing and operating such a system would introduce significant operational complexity and cost.

The Bottom Line

Can the backend metadata server (e.g., PostgreSQL) efficiently manage tables with billions of records and handle many concurrent readers and writers without becoming a performance bottleneck during metadata retrieval and query planning?

This remains to be battle-tested. With a well-tuned database on good-sized hardware and efficiently implemented indexing, clustering or partitioning (or even using distributed database backend) things could work out. But the important thing is to see and weigh out the trade-offs between the two metadata architectural patterns.

Having explored both architectural approaches in depth, let’s consolidate our understanding with a high-level comparison of key features and design choices.

High-level Feature Comparison

While this blog post is not about feature comparison between different open table formats, here is a high level comparison of some key features and architectural differences between DuckLake and other table formats:

As highlighted in the table above, DuckLake shares significant architectural similarities with Apache Hive while incorporating modern optimisations that address Hive’s shortcomings—specifically data file tracking, column-level statistics, improved concurrency management, and snapshot tracking.

With this technical foundation established, we can now address the central question: what does DuckLake’s future hold?

DuckLake’s Path Forward

It’s still too early to predict DuckLake’s fate. Just months after its launch, it faces fierce competition in a lakehouse market already dominated by established open table formats. The road ahead is challenging if DuckLake hopes to gain meaningful traction.

My personal assessment is that DuckLake has genuine potential, but not necessarily for large tech companies operating at massive scale. And that’s perfectly fine.

While MotherDuck will offer managed DuckLake at cloud scale, the open-source version could find strong adoption among small-to-medium self-hosted data lakes that never approach the metadata bottleneck of billions of records. After all, far more organisations manage small to moderate data volumes (< 100 TB) than operate at extreme scale.

Since DuckDB has already gained significant traction with strong ecosystem integration and support, DuckLake as a DuckDB extension can leverage this momentum.

Together, they could provide a lightweight Hybrid Analytical Lakehouse Storage (HALS) solution.

This represents a simple compelling unified analytics solution for anyone already using DuckDB in production who wants to extend its capabilities to managing data on data lakes.

With the rise of single-node processing paradigm, this solution extends that approach to manage unstructured and semi-structured data processing use cases, where storing data as Parquet files on object storage could deliver cost and performance benefits.

The Ecosystem Challenge

Beyond DuckDB’s territory, DuckLake’s success—particularly as an open-source portable and open table format—hinges entirely on community adoption.

So far in the months since its introduction, no significant contributions have emerged from outside DuckDB-related entities (DuckDB Labs, MotherDuck).

It’s important to note that, like Apache Iceberg, DuckLake is primarily a specification, not an engine. Just as Iceberg has implementations in Spark, Trino, and other engines, DuckLake can be implemented based on its published specification.

To reach its full potential, DuckLake spec requires broad adoption and integration throughout the data ecosystem. This includes support for popular Python data libraries like Polars, DataFusion, and Arrow; connectors for distributed processing engines such as Spark, Trino, and Flink; and data integration services like Fivetran, AWS Glue, dbt, and Airbyte.

Besides tooling, DuckLake requires genuine community interest and incentive to keep pace with the specification’s evolution as it matures over the coming years.

Without this backing, DuckLake risks remaining confined to the DuckDB ecosystem—limited to the DuckDB extension or MotherDuck’s managed cloud platform.

Whether the broader data community sees enough value to invest in DuckLake’s future remains the defining question.

Follow Me on LinkedIn

✨ Featured - Table Buckets as a New Lakehouse Abstraction

Since AWS announced S3 Tables last December, many industry experts have shared their opinions through blogs and social media. Most of the feedback I've read so far has been somewhat negative. Critics like Daniel argue that S3 Tables create vendor lock-in. They see it as a proprietary solution that limits flexibility and openness required by true Open Lakehouse architectures.

Critics also highlight that S3 Tables mainly integrate with AWS services such as Glue, Athena, and EMR, lacking sufficient support for general-purpose tools and services.

Despite these criticisms, broader support for S3 Tables is emerging. PyIceberg now supports accessing S3 Tables through Glue Catalog, and preview support for querying S3 Tables using Iceberg REST API endpoints, such as Amazon SageMaker Lakehouse REST endpoints, has been added to DuckDB.

In this context, Werner Vogels, Amazon's CTO, recently published another excellent article on his All Things Distributed blog. He reflects on the 19-year evolution of Amazon S3, from simple object storage to a comprehensive data solution. Vogels emphasises how customer feedback has shaped the development of S3's major capabilities.

Vogels further explains that S3's latest innovation, S3 Tables, directly addresses customer feedback and pain points experienced working with open table formats on S3 object store. Its main goal is to enhance lakehouse data management by resolving these challenges. Vogels also discusses the ongoing tension between simplicity and velocity, a common trade-off faced in product development. I highly recommend reading his article.

Read the full article here

📡 Open Source News

👉 Apache Flink 2.0.0 Release

Apache Flink 2.0.0 has been officially released, marking the first major Flink release since version 1.x launched nine years ago. Key innovations include disaggregated state management, materialised tables for unified stream-batch processing, and enhanced SQL capabilities. Check out the full highlights of new features and improvements. --> Read More

👉 Apache Kafka 4.0 Release

Apache Kafka 4.0 is officially out, marking a significant milestone by removing the dependency on Apache ZooKeeper. Key features highlighted by the Confluent blog include a new consumer group protocol for improved rebalance performance, the introduction of Queues for Kafka to support traditional queue semantics, updated Java version requirements, and various enhancements through Kafka Improvement Proposals (KIPs). --> Read More

👉 DuckDB Now Has Its Own Web UI!

From DuckDB v1.2.1 a local web user interface (UI) for DuckDB, has been developed in collaboration with MotherDuck, aimed at enhancing user experience by simplifying database interactions. Using the new UI you can now execute SQL queries through interactive notebooks, explore databases and columns with advanced features like syntax highlighting and autocomplete. --> Read More

🛠 Practical Data Engineering

👉 Implementation of Medallion Architecture Using ClickHouse

A great hands-on example of implementing a medallion architecture using ClickHouse to process and analyse free data from the Bluesky social network. It highlights common challenges such as data duplication, malformed JSON, and inconsistent structures. It also demonstrates how ClickHouse features—including the JSON data type and the ReplacingMergeTree engine—effectively address these issues. --> Read More

👉 DuckDB’s Enhanced Lakehouse Analytics

Querying external Delta Lake tables from DuckDB has become significantly easier with recent improvements in the Delta extension. You can now attach to Delta tables and query them using aliases in a simpler and cleaner manner. Additionally, data-skipping enhancements accelerate federated queries over open table formats, demonstrating DuckDB's commitment to advance federated analytics capabilities over lakehouse table formats. --> Read More

👉 21 Reasons to Consider Apache Hudi!

While Apache Iceberg has emerged as the leading open table format and continues strong into 2025, Apache Hudi is making its case by highlighting 21 unique reasons to consider Hudi over Iceberg and Delta Lake. If you're currently evaluating open table formats for your next project or company, this comparison is worth checking out. --> Read More

⚙️ Technical Deep Dive

👉 Parquet Data Skipping Mechanisms

This concise article provides an excellent overview of how pruning and data skipping are typically performed on Parquet files using metadata and statistics at various levels, including row groups and pages. The described approach aligns with DataFusion's implementation of the Parquet reading and pruning pipeline, but the general design principles apply broadly to other Parquet readers as well. --> Read More

👉 What's the Difference Between Arrow Flight, ADBC, and Arrow IPC?

Still confused about the differences between Arrow Flight Protocol, ADBC, and Arrow IPC? This article clearly explains these key technologies within the Apache Arrow ecosystem. It also highlights the advantages of using Apache Arrow as a data interchange format, demonstrating how it enables faster and more efficient data exchange compared to traditional methods. --> Read More

👉 Airflow Data Intervals: A Deep Dive

Understanding Apache Airflow's concept of time and intervals—such as start time, execution time, and logical dates—can be challenging. This article dives deep into the significance of data intervals in Airflow, clearly explaining their critical role in effectively scheduling and executing workflows. It illustrates how data intervals ensure each DAG run processes complete and accurate datasets, thus promoting idempotency and enabling reliable backfilling of historical data. --> Read More

📃 Academic Papers

👉 OLAP DBMS Archetype For The Next Ten Years?

An insightful paper co-authored by Michael Stonebraker and Andy Pavlo provides a comprehensive overview of the evolution of database management systems (DBMSs) since 2005. The authors predict the lakehouse architecture as the "OLAP DBMS archetype" for the coming decade, offering a unified infrastructure capable of supporting both SQL and non-SQL workloads—a vision initially conceived but not fully realised during the Hadoop era. --> Read More

💼 Data Engineering Career

👉 How Will AI Disrupt Data Engineering?

This is one of the most compelling questions facing software and data engineers today. Tristan Handy, founder and CEO of dbt Labs, shares insightful perspectives on how artificial intelligence will impact data engineering roles and what the future might hold for data engineers. --> Read More

💪 Skill Up

👉 Learning ClickHouse Fundamentals

ClickHouse is offering free 3-hour training sessions on ClickHouse fundamentals scheduled for April 22 and May 15. If you're interested, be sure to sign up! --> Read More

📣 Vendors News & Announcements

👉 General Availability of Confluent's TableFlow

Confluent has announced the general availability of TableFlow, a technology that embraces stream-table duality. TableFlow simplifies the ingestion of event data from Kafka topics into structured lakehouse tables (currently supporting Iceberg and Delta Lake), abstracting away complex data engineering tasks. It manages the entire ingestion lifecycle, including schema evolution and table maintenance on Confluent Platform. --> Read More

👉 Materialize Self-Managed & Free Community Edition

The Materialize streaming database service provider has introduced two new offerings: a Self-Managed version and a Free Community Edition. The Self-Managed option allows deployment of Materialize in your own environment, providing greater control over performance and compliance. The Community Edition grants free access to Materialize's powerful features with specific usage limits, making it easier to test or run small-scale production workloads. --> Read More

🔎 Case Studies

👉 Streaming Data Ingestion Into Cloud Data Warehouse

This insightful case study by Canva compares different data ingestion approaches into cloud data warehouses. It contrasts their previous micro-batch, file-based ingestion method (using AWS services like Firehose) with a new architecture using Snowflake's managed Snowpipe Ingestion service. The new solution achieves low-latency ingestion of billions of events daily, significantly reducing query latency to under 10 minutes and lowering overall cloud costs. --> Read More

👉 Using Key-Value Stores for Exactly-Once Streaming Ingestion

Event deduplication and exactly-once processing guarantees are crucial challenges in building reliable streaming pipelines, often complicated by pipeline failures and network issues. One popular solution involves using an external, fast key-value store to track and eliminate duplicate events during streaming. MyHeritage shares their experience implementing exactly-once processing using Spark Structured Streaming and a key-value store for deduplication. --> Read More

🎥 Conferences & Events

👉 Google Cloud Next '25 Keynotes

Google Cloud Next '25 took place earlier this month, and both the opening keynote and developer keynote are now available on YouTube. The conference again focused heavily on AI, highlighting Google's Gemini LLM foundation model, AI agents, the Agent Development Kit, and new features in Vertex AI. On the analytics front, the new capabilities of Google's Data Science Agent look particularly exciting. Be sure to check out these keynotes to stay updated on the latest trends and technologies from Google Cloud.

Opening Keynote:

Developer Keynote:

Follow Me on LinkedIn

The open source data engineering landscape continues to evolve rapidly, with significant developments across storage, processing, integration, and analytics in 2024.

This marks the second year the open source data engineering landscape is published. The goal is to identify and showcase key active projects and prominent tools in the data engineering space, and provide a comprehensive overview of the dynamic data engineering ecosystem, key trends and developments.

While this landscape is published annually, the accompanying GitHub repository is updated regularly throughout the year. Feel free to contribute if you notice any missing component.

Research Methodology

Conducting such extensive research demands considerable effort and time. I continuously research and strive to stay informed about significant developments in the data engineering ecosystem throughout the year, including news, activities, trends, reports, and advancements.

Last year, I built my own little data platform to track GitHub public repository events, enabling better analysis of GitHub-related metrics of open source tools such as code activity, stars, user engagement, and issue resolution.

The stack includes a data lake (S3), Parquet as the serialisation format, DuckDB for processing and analytics, Apache NiFi for data integration, Apache Superset for visualisation, and PostgreSQL for metadata management, among other tools. This setup has allowed me to collect approximately 1TB of raw GitHub event data, consisting of billions of records, along with an aggregated dataset that rolls up daily, totaling over 500 million records for 2024.

Tool Selection Criteria

The available open source projects for each category are obviously vast, making it impractical to include every tool and project in the presented landscape.

While the GitHub page contains a more comprehensive list of tools, the annually published landscape only contains active projects, excluding inactive and fairly new projects with no minimal maturity or traction. However not all included tools may be fully production-ready; some are still on their journey toward maturity.

Without further ado, here is the 2025 Open Source Data Engineering Landscape:

State of Open Source in 2025

The open source data engineering ecosystem experienced substantial growth in 2024, with over 50 new tools added to this year's landscape while removing approximately 10 inactive and archived projects. Although not all these tools launched in 2024, they represent important additions to the ecosystem.

While this growth demonstrates continued innovation, the year also saw some concerning developments regarding licensing changes. Established projects including Redis, CockroachDB, ElasticSearch, and Kibana transitioned to more closed and proprietary licenses, though Elastic later announced a return to open source licensing.

However, these shifts were balanced by significant contributions to the open source community from major industry players. Snowflake's contribution of Polaris, Databricks' open sourcing of Unity Catalog, OneHouse's donation of Apache XTable, and Netflix's release of Maestro demonstrated ongoing commitment to open source development from industry leaders.

The Apache Foundation maintained its position as a key steward of data technologies, actively incubating several promising projects throughout 2024. Notable projects in incubation included Apache XTable (universal table format), Apache Amoro (lakehouse management), Apache HoraeDB (time-series database), Apache Gravitino (data catalog), Apache Gluten (Middleware), and Apache Polaris (data catalog).

The Linux Foundation has also strengthened its position in the data space, continuing to host exceptional projects such as Delta Lake, Amundsen, Kedro, Milvus, and Marquez. The foundation expanded its portfolio in 2024 with new significant additions, including vLLM, donated by the University of California, Berkeley, and OpenSearch, which was transferred from AWS to the Linux Foundation.

Open Source vs Open Core vs Open Foundation

Not all of the projects listed are fully interoperable, vendor-neutral open source tools. Some operate under an open core model, where not all components of the complete system are available in the open source version. Typically, critical features such as security, governance, and monitoring are reserved for the paid versions.

Questions remain about the sustainability of the open core business model. This model faces significant challenges, leading some to believe it may give way to the Open Foundation model. In this approach, open source software serves as the backbone of commercial offerings, ensuring that it remains a fully viable product for production with all the necessary features.

Overview of Categories

The data engineering landscape is divided into 9 major categories:

1. Storage Systems: Databases and storage engines spanning OLTP, OLAP, and specialised storage solutions.

2. Data Lake Platform: Tools and frameworks for building and managing data lakes and lakehouses.

3. Data Processing & Integration: Frameworks for batch and stream processing, plus Python data processing tools.

4. Workflow Orchestration & DataOps: Tools for orchestrating data pipelines and managing data operations.

5. Data Integration: Solutions for data ingestion, CDC (Change Data Capture), and integration between systems.

6. Data Infrastructure: Core infrastructure components including container orchestration and monitoring.

7. ML/AI Platform: Tools focused on ML platforms, MLOps and vector databases.

8. Metadata Management: Solutions for data catalogs, governance, and metadata management.

9. Analytics & Visualisation: BI tools, visualisation frameworks, and analytics engines.

In the following section latest trends, innovations and current state of major products in each category is briefly discussed.

1. Storage Systems

The storage systems landscape has seen significant architectural advancements in 2024, particularly in the realm of OLAP database systems.

DuckDB emerged as a major success story, particularly following its 1.0 release that demonstrated production readiness for enterprise use. The new embeddable OLAP category has expanded with new entrants like chDB (built on ClickHouse), GlareDB, and SlateDB, reflecting growing demand for lightweight analytical processing capabilities.

OLAP Extensions & HTAS

A significant development has been the proliferation of new OLAP extensions, especially in the PostgreSQL ecosystem.

These extensions allow to seamlessly extend OLTP databases, transforming these systems into HTAP (Hybrid Transactional/Analytical Processing) or new HTAS (Hybrid Transactional Analytical Storage) database engine that integrate headless data storage—like data lakes and lakehouses—with transactional database systems.

MotherDuck's release of pg_duckdb represented a major advancement, enabling DuckDB to serve as an embedded OLAP engine within PostgreSQL. The pg_mooncake extension followed, providing native column store capabilities in open table formats like Iceberg and Delta. Crunchy Data and ParadeDB made similar contributions through pg_parquet and pg_analytics respectively, enabling direct analytics over Parquet files on data lakes.

Zero-Disk Architecture

The zero-disk architecture emerged as perhaps the most transformative trend in storage systems, fundamentally changing how database systems manage storage and compute layers.

This architectural approach completely eliminates the need for locally attached disks, instead using remote deep storage solutions like S3 object storage as the primary persistence layer.

Beyond OLAP storage systems, such as cloud data warehouses and open table formats, we are witnessing a significant emergence of this pattern in NoSQL, real-time, streaming and transactional systems.

The primary trade-off for disk-based vs zero-disk systems is cost vs performance, and the I/O latency for reading and writing data to the physical storage. While disk-based systems can manage fast sub-millisecond I/O, the zero-disk systems achieve economics of scale with cheap scalable object storage, at the cost of facing latencies up to one second when reading and writing data to a an object storage service.

New database systems including SlateDB and Apache HoraeDB time-series database were built from the ground up with this architecture, while established systems like Apache Doris and StarRocks adopted it in 2024. Other real-time engines such as AutoMQ and InfluxDB 3.0 are increasingly adopting the zero-disk paradigm.

For a comprehensive analysis of zero-disk architecture and its implications, see the detailed exploration in the following article:

Other Notable Developments

Following Redis's move to a proprietary license in 2024, Valkey emerged as a leading open source alternative, becoming the most-starred storage system on GitHub in 2024. Major cloud providers quickly adopted it, with Google integrating it into Memorystore and Amazon supporting it through ElastiCache and MemoryDB services.

Other notable developments include ParadeDB, an alternative to Elasticsearch built on the PostgreSQL engine, and new hybrid streaming storage systems like Proton from TimePlus and Fluss introduced by Ververica. These systems aim to integrate streaming and OLAP functionalities with a columnar storage foundation.

2. Data Lake Platform

With database pioneer Michael Stonebraker endorsing the lakehouse architecture and open table formats as 'the OLAP DBMS archetype for the next decade', data lakehouse continues to be the hottest topic in data engineering.

The open table format landscape continued to evolve significantly in 2024. The forth major open table format, Apache Paimon graduated from incubation, bringing streaming lakehouse capabilities with Apache Flink integration. Apache XTable emerged as a new project focused on bi-directional format conversion, while Apache Amoro entered incubation with its lakehouse management framework.

In 2024, Apache Iceberg has established itself as the leading project among open table format frameworks, distinguished by its ecosystem expansion and GitHub repository metrics, including a higher number of stars, forks, pull requests, and commits.

All major SaaS and cloud vendors have been enhancing their platforms to support access to open table formats. However, write support has been less prevalent, with Apache Iceberg being the preferred choice for comprehensive CRUD (Create, Read, Update, Delete) integration.

Google's BigLake Managed Tables, enabling mutable Iceberg tables within customer-managed cloud storage, Amazon's newly announced S3 Tables with native Iceberg support, and other major SaaS tools such as Redpanda launching Iceberg Topics and the Crunchy Data Warehouse deeply integrating with Apache Iceberg, are examples of increasing adoption and deep integration with Iceberg in the ecosystem.

Going forward universal table formats like Apache XTable and Delta UniForm (Delta Lake Universal Format) may face significant challenges in navigating the potential divergence of features across various formats, and the fate of open table formats may mirror that of open file formats, when Parquet emerged as the de facto standard.

As the lakehouse ecosystem continues to grow, the adoption of interoperable open standards and frameworks within an Open Data Lakehouse platform is expected to gain more popularity.

Emergence of Native Table Format Libraries

A new trend is emerging in the lakehouse ecosystem focused on developing native libraries in Python and Rust. These libraries aim to provide direct access to open table formats without the need for heavy frameworks like Spark.

Notable examples include Delta-rs, a native Rust library for Delta Lake with Python bindings; Hudi-rs, a Rust implementation for Apache Hudi with a Python API, and PyIceberg, an evolving Python library designed to enhance accessibility to the Iceberg table format outside the default Spark engine.

3. Data Processing & Integration

Rise of Single-Node Processing

The rise of single-node processing represents a fundamental shift in data processing, challenging traditional distributed-first approaches.

Recent analyses show that many companies have overestimated their big data needs, prompting a reassessment of their data processing requirements. Even in the organisations with large data volumes, approximately 90% of queries remain within manageable workload size to run on a single machine, only scanning recent data.

Modern single-node processing engines, such as DuckDB, Apache DataFusion, and Polars, have emerged as powerful alternatives, capable of handling workloads that previously necessitated distributed systems like Hive/Tez, Spark, Presto or Amazon Athena.

To explore the comprehensive analysis on the state of single-node processing, please follow the link below:

Stream Processing

The stream processing ecosystem continued to expand in 2024, with Apache Flink further solidifying its position as the premier streaming engine, while Apache Spark retains it’s strong position.

Celebrating its 10th anniversary, Flink released version 2.0, representing the first major update since Flink 1.0 debuted eight years ago. The Apache Flink ecosystem expanded significantly with the introduction of the Apache Paimon open table format and newly open sourced Fluss streaming engine. In 2024, leading cloud vendors have increasingly integrated Flink into their managed services, latest being Google’s serverless BigQuery Engine for Apache Flink solution.

Emerging streaming engines are Fluvio, Arroyo and FastStream, striving to compete with these established contenders. Fluvio and Arroyo stand out as the only Rust-based engines which aim to eliminate the overhead typically associated with traditional JVM-based stream processing engines.

In major open source streaming news, Redpanda acquired Benthos.dev, rebranding it as Redpanda Connect and transitioning it to a more proprietary license. In response, WarpStream forked the Benthos project, renaming it Bento and committing to keeping it 100% MIT-licensed.

Python Processing Frameworks

In the Python data processing ecosystem Polars is currently the dominant high-performance DataFrame library for data engineering workloads (excluding PySpark). Polars achieved an impressive 89 million downloads in 2024, marking a significant milestone with its 1.0 release.

However, Polars now faces competition from DuckDB's DataFrame API, which has captured the community's attention with its remarkably simple integration with external storage systems and zero-copy integration (direct memory sharing between different systems) with Apache Arrow—similar to Polars. Both libraries rank in the top 1% of the most downloaded Python libraries last year.

Apache Arrow has solidified its position as the de facto standard for in-memory data representation in the Python data processing ecosystem. The framework has established deep integration with various Python processing frameworks including Apache DataFusion, Ibis, Daft, cuDF, and Pandas 3.0.

Ibis and Daft are other innovative DataFrame projects with high potential. Ibis features a seamless back-end interface to various SQL-based databases and Daft provides distributed computing capabilities, built from the ground up to support distributed DataFrame processing.

4. Workflow Orchestration & DataOps

In 2025, open source workflow orchestration category continues to stand as one of the most dynamic segments of the data engineering ecosystem, featuring over 10 active projects that range from established platforms like Apache Airflow to newly open sourced engines like Netflix's Maestro.

After a decade Apache Airflow continues to be the most deployed and adopted workflow orchestration engine with a staggering 320M downloads in 2024 alone, while facing competition from rising competitors such as Dagster, Prefect and Kestra.

Interestingly, Kestra gained the most stars on GitHub in 2024, with a surge directly linked to its $8M funding announcement in September, which was featured on TechCrunch. In terms of code activity, Dagster demonstrated remarkable development activity with an impressive 27K commits and close to 6K pull requests closed in 2024.

For comprehensive analysis on the state of workflow orchestration systems, read the following article:

Data Quality

Great Expectations continues to be a leading Python framework for data quality and validation also featured in Databrick's Top 10 Data and AI productions of 2024 , followed closely by Soda and Pandera in the data engineering practice. However, there is some disappointing news: the Data-Diff project has been archived by its main maintainer, Datafold in 2024.

Data Versioning

Data Versioning remains a prominent topic in 2024, as efforts continue to bring the capabilities of modern version control systems, like Git, to data lakes and lakehouses.

Projects like LakeFS and Nessie, enhance modern data lakes and open table formats such as Iceberg and Delta Lake by extending their transactional metadata layers.

Data Transformation

The scope of using dbt for data transformation is expanding beyond its original focus on data modeling within data warehouse systems. It is now making inroads into off-warehouse environments, such as data lakes, through new integrations and plugins that leverage ephemeral compute engines like Trino.

Currently, dbt faces competition primarily from SQLMesh. A notable stand-off in 2024 was the SQLMesh vs. dbt debate, highlighted by Tobiko's CEO, who claimed on social media that SQLMesh is so good it's banned from dbt's Coalesce conference!

5. Data Integration

In the data integration space, Airbyte maintained its leadership position, achieving an impressive milestone by closing 13K pull requests in preparation for version 1.x. The dlt framework demonstrated significant maturation with its 1.0 release, while Apache SeaTunnel gained traction as a compelling alternative.

The Change Data Capture (CDC) framework landscape evolved with new tools including Artie Transfer and PeerDB (acquired by ClickHouse), while Flink CDC connectors gaining adoption among platforms using Flink as their primary streaming engine.

Event Hubs (Streaming Pub/Sub Services)

One of the most notable innovations in the data integration space in 2024 came from the evolving data streaming landscape. A significant architectural shift in this category is the separation of storage and compute, coupled with the adoption of object storage in a zero-disk architecture. WarpSteram is a pioneer of implementing this architecture in real-time streaming space.

This model also enables a flexible Bring Your Own Cloud (BYOC) deployment strategy, as both compute and storage can be hosted on the customer's preferred infrastructure, while the service provider maintains the control plane.

WarpStream's success has prompted major competitors to adopt similar architectures. Redpanda launched Cloud Topics, enhancing its offerings, while AutoMQ implemented a hybrid approach featuring a fast caching layer to improve I/O performance.

Additionally, StreamNative introduced the Ursa engine for Apache Pulsar, and Confluent unveiled its own cloud-native Freight Clusters in 2024. Ultimately, Confluent decided to acquire WarpStream, further expanding its offering with BYOC model. Meanwhile, the remarkable Apache Kafka stands at a crossroads that may define its future direction in the ecosystem.

6. Data Infrastructure

The data infrastructure landscape in 2024 remained largely stable, with Kubernetes celebrating its 10th anniversary while maintaining its position as the leading resource scheduling and virtualisation engine in cloud environments.

In the observability space, InfluxDB, Prometheus, and Grafana continued their dominance, with Grafana Labs securing a notable $270M funding round that reinforced the long-term viability of their core products like Grafana as general-purpose observability solutions.

7. ML/AI Platform

Vector databases maintained strong momentum from 2023, with Milvus emerging as a leader alongside Qdrant, Chroma, and Weaviate. The category now encompasses ten active vector database projects, reflecting the growing importance of vector search capabilities in modern AI-enabled data architectures.

The introduction of LLMOps (also referred to as GenOps) as a distinct category in this year's presented landscape was marked by the rapid growth of new projects like Dify and vLLM purposefully built for managing LLM models.

8. Metadata Management

Metadata management platforms have gained significant momentum in recent years, with DataHub leading the open source space through its active development and community engagement.

However, the most notable developments in 2024 occurred in catalog management. While 2023 was dominated by competition in open table formats, 2024 marked the beginning of the Catalog War.

In contrast to earlier years, 2024 brought a wave of new open catalog solutions to the market, including Polaris (open sourced by Snowflake), Unity Catalog (open sourced by Databricks), LakeKeeper, and Apache Gravitino.

This proliferation reflects the realisation that emerging data lakehouse platforms, which rely heavily on open table formats, lack advanced built-in catalog management capabilities for seamless multi-engine interoperability.

All of these projects have the potential to establish a new standard for vendor-agnostic, open catalog services in data lakehouse platforms. Much like Hive Metastore became the de facto standard for Hadoop-based platforms, these emerging catalogs may finally replace Hive Metastore's long-standing dominance in catalog management on open data platforms.

9. Analytics & Visualisation

In the open source Business Intelligence realm, Apache Superset and Metabase remain the leading BI solutions. While Superset leads in GitHub popularity, Metabase shows the highest development activity. Lightdash emerged as a promising newcomer, securing $11 million in funding and demonstrating market demand for lightweight BI solutions.

BI-as-Code Solutions

BI-as-Code emerged as a distinctive category, driven by the continued success of Streamlit, which maintained its position as the most popular BI-as-Code solution.

These tools enable developers to create interactive apps and lightweight BI dashboards using code, SQL and templates like Markdown or YAML, being able to combine the software engineering best practices, such as version control, testing and CI/CD into the dashboard development workflow.

In addition to Streamlit and the well-known Evidence, new entrants like Quary and Vizro have gained traction, with Quary notably implementing a Rust-based approach that diverged from the Python-centric norm of the category.

Composable BI Stack

The evolution of system decomposition is not limited to storage systems; it has also impacted Business Intelligence (BI) stacks. A new trend is emerging that combines lightweight, bottomless BI tools (which don't have a back-end server) with headless embeddable OLAP solutions such as Apache DataFusion, Apache Arrow, and DuckDB.

This integration addresses several gaps in the the open source BI stack such as native ability to query external data lakes and lakehouses while preserving the benefits of lightweight, disaggregated architectures.

BI Products like Omni, GoodData, Evidence, and Rilldata have already incorporated these engines into their BI and data exploration tools. Both Apache Superset (using the duckdb-engine library) and Metabase now support embedded DuckDB connections.

For a comprehensive analysis of the evolving composable BI architecture, see the detailed exploration in the following article:

MPP Query Engines

Post-Hadoop era there has been little innovation and introduction of new open source MPP (Massively Parallel Processing) systems while existing engines continue to mature.

While Hive's share is shrinking, Presto and Trino still remain as top open source MPP query engines used in production, despite facing fierce competition from Spark as a unified engine, and managed cloud MPP products such as Databricks, Snowflake and AWS Redshift Spectrum plus Athena.

Future Outlook and Conclusion

The open source data ecosystem is entering a phase of maturity in key areas such as data lakehouse, characterised by consolidation around proven technologies and increased focus on operational efficiency.

The landscape continues to evolve toward cloud-native, composable architectures while standardising around dominant technologies. Key areas to watch include:

• Further consolidation in the open table format space

• Continued evolution of zero-disk architectures in real-time and transactional systems

• Quest toward providing a unified lakehouse experience

• The rise of LLMOps and AI Engineering

• The expansion of the data lakehouse ecosystem in areas such as open catalog integration and development of native libraries

• The increasing traction of single-node data processing and embedded analytics

Follow Me on LinkedIn

This is the fifth part in the Data Landscape Trends 2024-2025 series, focusing on the state of the open source workflow orchestration systems.

Introduction

In the rapidly evolving landscape of data engineering, workflow orchestration engines play a key role in managing complex data processes.

This analysis explores the current state of workflow orchestration engines through multiple lenses such as community engagement, technical architecture, adoption metrics, and emerging innovations in 2024.

We'll cover the following topics:

• Current open source workflow orchestration landscape

• Open Source vs Open Core engines

• Task-centric vs Data-centric engines

• GitHub repository trends in 2024

• Summary and analysis of the current state of the products

• Major 2024 workflow orchestration trends

• Recommendations and conclusion

Current OSS Landscape and Major Products

The workflow orchestration category stands out as one of the most dynamic segments of the open source data engineering ecosystem.

It features over 10 active projects that range from established products like Apache Airflow to newly open sourced engines like Netflix's Maestro.

The evolution of major open source workflow orchestration engines traces back to 2008, when Yahoo developed the first significant workflow engine Oozie to address the growing complexity of managing workloads on the Hadoop platform.

Since then, the industry has developed numerous orchestration systems to meet the growing demands of workload management and orchestration on data platforms.

Some projects, such as Orchest, have been come and gone, and are no longer maintained. Such retired projects are excluded from this analysis.

The timeline below illustrates the development progression of major open source workflow orchestration engines, highlighting both their initial open source releases and subsequent donations to the open source foundation where applicable.

I remember back in 2018 when we had to pick a workflow engine for a new large-scale data platform. Our options were pretty much just Luigi, Azkaban, and Airflow.

The choice was really simple back then - Airflow was a clear winner since it was Python-based and had great features. But nowadays it's so much harder to navigate this landscape and do a proper comparison of architectures and features between all the available tools in the market.

Netflix's New Contribution

An exciting development in this ecosystem came when Netflix open sourced their next-generation orchestrator, Maestro, in July 2024.

Introduced via their tech blog, Maestro is designed as a highly scalable and flexible scheduler capable of handling large-scale heterogeneous workflows like ML training and data pipelines.

What makes Maestro stand out is its flexible execution support for Docker images and notebooks, along with its ability to handle both cyclic and acyclic (DAG) workflow patterns.

Since its July release, Maestro has gained notable traction in the community. However, the repository has seen limited code activity since the initial release.

Back-end Language

In terms of back-end languages, these tools have a fairly even distribution between Java, Go, and Python, with the exception of Windmill, which is built using the rising Rust language.

Open Source vs Open Core Engines

It's important to note that not all these projects are truly open source. Some follow an "open core" model instead, where the main SaaS provider only releases certain core components as open source while keeping premium features such as monitoring and security, proprietary.

When evaluating these tools for adoption, it's crucial to assess how portable and genuinely open source each project really is, as this can impact long-term sustainability and cost.

Many current open core tools like Kestra and Dagster keep essential enterprise features – especially security features like SSO – locked behind their enterprise versions. This is a deliberate strategy to monetise enterprise clients who need these capabilities.

This approach creates a significant problem for OSS adoptions: businesses that care about security and governance can't realistically use the open source versions of these products.

Open Core users frequently complain about this limitation, particularly the lack of basic authentication and authorisation mechanisms in the open core versions.

Currently, only projects like Apache Airflow, Flyte and Apache DolphinScheduler are guaranteed to remain fully open source, as they're not owned by any single commercial entity but rather governed by an open source Foundation.

Task-Centric vs Data-Centric Engines

Workflow orchestration engines can be broadly classified by their fundamental approach to workflow management: task-centric versus data-centric, alongside other categories like declarative vs code-based and batch vs event-driven.

Task-Centric Orchestrators

Airflow, Luigi, Cadence, and Kestra exemplify the task-centric approach, organising workflows as Directed Acyclic Graphs (DAGs) of interconnected tasks.

In these engines, the task is the primary unit of work, capable of executing any type of operation. The scheduler's main concern is managing control flow and dependencies between tasks within the DAG, remaining largely agnostic to the actual work being performed.

Data-Centric Orchestrators

Engines like Dagster, Temporal, and Flyte take a more opinionated, data-centric approach. In these engines, data-oriented objects (or "assets" in Dagster's terminology) serves as the primary focus of the workflow.

They treat workflows as data-aware pipelines where assets - whether tables, files, ML models, or dbt models - are produced, consumed, and transformed.

Data-centric engines provide native support for passing data between tasks and offer superior integration with modern data transformation frameworks like dbt and SQLMesh, compared to task-centric engines.

GitHub Repository Trends

Open source projects are typically evaluated through key metrics including GitHub stars, download counts, contributor activity, and repository engagement (measured by commits, releases, and issue resolution rates).

As part of my commitment to understanding the open source ecosystem, I've developed my own small analytical platform that tracks and analyses all GitHub events for public repositories. The following metrics and trends for 2024 are derived from this platform.

Project Popularity

Looking at GitHub repository star trends in 2024, Kestra has emerged as a rising workflow orchestration project. The graph below shows a spike in September, when Kestra surpassed all other projects in new stars gained in 2024.

This surge is directly linked to Kestra's $8M funding announcement, which was featured in TechCrunch. It's a clear example of how repository stars can spike in response to major company announcements.

The well-established Apache Airflow and Prefect ranked the second and third most-starred workflow projects in 2024 respectively.

Code Activity

Code activity in open source projects can be measured by two main metrics: pull requests (opened, closed, and reviewed) and commit volume (push events).

For 2024, Dagster and Airflow led the pack in pull request activity, each processing over 10K PRs from their contributors, with Prefect following close behind.

On the other end of the spectrum, projects like Cadence, Luigi, Maestro, and Azkaban showed concerning levels of inactivity, raising questions about their long-term health.

Looking at commit volume, Dagster demonstrated remarkable development activity with an impressive 27K commits in 2024. Prefect and Windmill also showed strong development momentum, each recording over 10K commits.

Project Collaboration

The health and sustainability of an open source project largely depends on its contributor base – the wider and more diverse, the better.

When evaluating contributor metrics, it's crucial to distinguish between active contributors who consistently work throughout the year and one-off contributors who make occasional submissions. Active contributors provide a more meaningful measure of project health.

Looking at active contributors in 2024, Airflow and Dagster lead the ecosystem with over 20 active contributors each. Any major open source project with few (ex < 5) active contributors raises sustainability concerns. By this metric, projects like Argo Workflows, Mage-ai, DolphinScheduler, and Flyte fall into a warning zone.

At the concerning end of the spectrum, projects like Luigi and Azkaban showed no active contributions throughout the year.

Community Engagement

Community engagement can be measured through several indicators: issues logged, comment volume, and participation in official community channels like Slack and discussion boards. These metrics help determine how vibrant and active a tool's community really is.

Another key metric is the ratio of closed to opened issues, which indicates how quickly project maintainers address community-reported problems.

Looking at GitHub activity in terms of total issues opened and closed, Airflow, Kestra, Prefect, and DolphinScheduler show the strongest community engagement.

Based on total issues registered, We can consider fewer than 100 issues or less than 50% issue resolution a concern, and fewer than 50 issues a danger zone. Again, projects like Luigi, Azkaban, and Cadence fall into this danger zone, suggesting minimal community interaction.

Downloads & Installations

Most open source orchestration tools are either Python-based or provide Python client and SDKs, making PyPI download statistics a useful metric for measuring adoption and popularity.

Looking at the download stats from clickpy.clickhouse.com, Apache Airflow dominates the ecosystem with a staggering 320M downloads in 2024 alone - ten times more than its nearest competitor. This reinforces Airflow's position as the leading tool in the entire data engineering ecosystem.

Prefect and Dagster round out the top three most downloaded packages in 2024, with 32M and 15M downloads respectively.

An interesting observation: despite being an inactive project, Luigi recorded 5.6M downloads in 2024. This likely reflects existing users updating to minor releases, suggesting a significant legacy user base still relies on the platform.

Summary & Analysis

Here is the summary of the evaluation of the workflow orchestration engines across key GitHub metrics:

Advancing Projects

After a decade, Airflow remains the dominant force in open source orchestration, maintaining the most active and vibrant open source project in the market.

Dagster is likely the second most popular orchestrator in 2024. Along with Prefect and Temporal, it's capturing significant market attention, particularly among startups and smaller-scale deployments.

These tools stand out for their simplified approach to data-centric workflow management, more intuitive UIs, and enhanced support for event-driven workflows.

Rising Projects

Kestra has become one of the fastest-growing orchestration tools in 2024, gaining momentum after securing $8M in funding. The project has also been praised for its simplicity, declarative YAML-based workflow definitions, and support for event-driven workflows.

Declining Projects

Legacy tools Luigi and Azkaban rank at the bottom across all metrics. While neither project has been officially archived or retired, their lack of meaningful development activity in 2024 effectively marks them as inactive.

Luigi saw only minor bug fixes throughout the year, while Azkaban showed no code activity whatsoever. This dramatic decline in maintenance suggests these once-popular orchestrators have reached the end of their active lifecycle.

The future of Netflix's Maestro remains uncertain. 2025 will be a pivotal year, revealing whether the project gains momentum on GitHub or follows the path of some other abandoned in-house tools released by tech giants.

2024 OSS Orchestration Competition

Let's turn this into a competition and rank our open source workflow engines!

We'll identify the top three performers across key metrics in 2024, creating a sort of "workflow orchestrator competition."

Based on our OSS metrics and medal counts, Apache Airflow claims the crown as 2024's champion workflow orchestrator, with Dagster taking second and Prefect earning third place.

Important note: This comparison focuses solely on open source activity metrics and community engagement. It should not be interpreted as a judgment of each tool's features, capabilities, or overall ecosystem. The best workflow orchestrator for your needs will depend on your specific requirements, use cases, and technical environment.

Major 2024 Trends

Let’s explore the key development trends in the workflow orchestration ecosystem for 2024.

Event-Driven & Real-Time Orchestration

The workflow orchestration ecosystem is shifting toward event-triggered and real-time processing capabilities, reflecting industry's growing demand for real-time workload management.

In 2024, several major products made significant moves in this direction. Kestra introduced Real-time and HTTP Triggers, enabling millisecond-latency responses to events from systems like Kafka and AWS SQS, and over HTTP requests.

Temporal enhanced its real-time capabilities with Workflow Update and Workflow Update-With-Start features, enabling synchronous processing for interactive applications. Meanwhile, DolphinScheduler expanded its event-driven architecture with a variety of new triggers.

Mage focused on real-time data processing, introducing Streaming Pipelines that support real-time ingestion and transformation from sources like Kafka and Google Pub/Sub.

Even Apache Airflow, traditionally a batch-oriented system, has recognised this shift toward real-time processing. Its 2024 updates introduced addition of new conditions for its data-aware scheduling, and new scheduling mechanism which supports scheduling DAGs based on both dataset events and time.

AI/LLM Integration & Automation

The integration of AI and LLM capabilities emerged as another major trend in workflow orchestration during 2024, reflecting the growing role of LLM-based workloads in enterprise data operations.

Prefect made a significant move in this space by launching ControlFlow, a framework specifically designed for AI-driven workflows and LLM integration. Prefect also integrated Marvin, an LLM-powered assistant, to simplify the creation of AI workflows.

Temporal embraced multi-agent workflows, enabling sophisticated coordination between AI models, software applications, and human participants.

Meanwhile, Windmill took a different approach by integrating AI directly into the development experience, introducing an AI copilot to assist in flow building.

Enhanced Resource Management & Execution

Intelligent resource management has become a critical focus for workflow engines, particularly as organisations increasingly run workflows on cloud-managed and serverless platforms. Several cloud-native engines made significant advances in this area during 2024.

Temporal introduced sophisticated resource management with its worker auto-tuning feature, which automatically adjusts worker slots based on real-time CPU and memory usage.

Kestra has introduced task runners that can dynamically offload resource-intensive tasks to on-demand compute services like Azure Batch, Google Batch, and Google Cloud Run.

Dagster Pipes became stable in version 1.8 released in 2024, with enhanced integrations for Lambda, Kubernetes, and Databricks looking ahead.

DolphinScheduler plans to integrate KEDA (Kubernetes Event-Driven Autoscaling), which will enable automatic worker scaling based on workload demands, further enhancing its Kubernetes-native capabilities.

Prefect and Flyte expanded their back-end execution capabilities in 2024 by enhancing support for distributed computing frameworks, integrating with scalable Python execution frameworks such as Ray and Dask, enabling more efficient parallel processing and distributed task execution.

Conclusion & Recommendations

After a decade, Apache Airflow remains the most mature and widely adopted orchestration tool in the data engineering ecosystem. Its position as the market leader is reinforced by major cloud vendors - Google Cloud Composer and Amazon MWAA have both standardised on Airflow for their managed workflow services.

While Airflow faces criticism for its steep learning curve, operational overhead, and not-so friendly UX with outdated UI (though a complete revamp is planned for the upcoming version 3.0), its primary technical limitation is its focus on batch-oriented workflows, with less native support for modern dynamic workflow patterns.

For large-scale deployments managing large number of heterogeneous workflows that require a general-purpose engine with extensive operations support and a large ecosystem, Apache Airflow remains the top choice. At the Airflow Summit 2024, major companies showcased Airflow's massive scalability, with Uber orchestrating 450K pipeline runs daily across 1000 teams, Stripe managing 150K tasks, and LinkedIn operating over 10K parallel DAGs.

For startups, and small to mid-sized businesses consider newer orchestration tools that offer streamlined setup and development experience through features like in-browser development environments, declarative workflow authoring, and low-code capabilities.

For dynamic and data-centric workflow orchestration, products like Prefect and Dagster excel at data-aware orchestration compared to traditional task-based schedulers.

Follow Me on LinkedIn

This is the forth part in the Data Landscape Trends 2024-2025 series, focusing on the state of the open-source real-time OLAP database systems.

In the first part, we explored the evolution of the BI stack; the second part examined the rise of single-node processing engines; and the third part discussed the evolution of zero-disk architecture.

Introduction

Real-time OLAP database systems have undergone significant development since their early development in the 2010s by companies like Yandex, Metamarkets and LinkedIn, which introduced systems such as ClickHouse, Druid, and Pinot to address the limitations of traditional MPP engines.

Initially designed for sub-second analytics on massive volumes of append-only web logs and clickstream data, these specialised databases have expanded their capabilities over the years.

New entrants like Apache Doris and StarRocks have joined the scene, aiming to bridge the gap between traditional OLAP architectures and modern MPP systems.

This article provides a comprehensive overview of the evolving real-time OLAP ecosystem, examining the current state and advancements of major open-source OLAP engines. We'll explore:

• Background on real-time OLAP database systems

• An analysis of the current landscape and leading open-source products

• Emerging trends in 2024

• Major features and capabilities introduced by each product in 2024

• An assessment of key open source metrics including popularity, development activity, and community engagement

• A comprehensive comparison of architectural approaches and core features

• Recommendations for choosing and implementing these systems

Real-time OLAP Database Systems

Real-time OLAP engines are specialised databases designed to deliver sub-second analytics performance.

They operate similarly to cube servers in traditional BI solutions, pre-computing metrics across various dimensions to enable real-time drilling and slice-and-dice analysis of data.

These engines achieve exceptional query performance through a combination of optimised storage and sophisticated indexing during data ingestion. Their architecture treats data as immutable, optimised primarily for append-only operations and segment-level (i.e. data chunks) replacements.

Trade-offs

While this storage paradigm is particularly well-suited for log and event workloads in denormalised format, it comes with certain trade-offs: inflexible data models, limited join capabilities, and higher ingestion latency.

Unlike modern MPP-based scalable storage systems such as Redshift, BigQuery, and Snowflake—which couldn't deliver sub-second queries due to architectural limitations—these systems prioritised query performance over conventional database features like table joins, ACID guarantees, and row-level mutations.

In recent years, ClickHouse and newer systems like Apache Doris and StarRocks are moving beyond the traditional real-time OLAP storage model by introducing support for mutable data operations and complex queries typically associated with data warehouse systems.

Current Landscape and Major Products

The following graph illustrates the development timeline of major open-source real-time OLAP engines, highlighting when each system was open-sourced and, where applicable, donated to the Apache Software Foundation.

The real-time OLAP landscape features several established products, each with its own origin story. ClickHouse was initiated at Yandex in 2010, followed by Apache Druid (developed by Metamarkets in 2011), Apache Kylin (created by eBay), and Apache Pinot (originated at LinkedIn in 2013).

More recently, the ecosystem expanded with two significant additions: Apache Doris and StarRocks. Baidu developed and open-sourced Apache Doris in 2017-2018.

StarRocks later emerged as a fork of Doris, led by former Apache Doris PMC members who sought to address what they perceived as gaps in the original project's roadmap for building modern real-time analytics systems.

Divergence From Traditional OLAP Model

While Apache Doris and StarRocks are classified as real-time OLAP systems, they represent a divergence from traditional OLAP approaches.

Rather than relying on immutable storage models and heavy pre-processing ad indexing methods with cube semantics, these systems bridge the gap between conventional OLAP designs and modern MPP (Massively Parallel Processing) architectures like Amazon Redshift and Google BigQuery.

Through native support for both bulk and row-level updates, alongside complex join capabilities, these systems are aiming to offer a hybrid solution that combines real-time OLAP performance with the versatility of general-purpose analytics platforms.

The ecosystem also includes proprietary solutions such as Rockset and Kinetica, though these fall outside the scope of this open-source focused analysis.

GitHub Repository Trends

Open-source projects are often evaluated based on key metrics such as repository stars, download counts, contributor activity, and repository engagement, including commits, releases and issues logged and resolved.

As part of my work and passion for the open-source ecosystem, I run my own little analytics platform to collect, store, and analyse GitHub events to track year-over-year trends in data engineering-related projects.

Project Popularity

By using GitHub's Watch (Star) and Fork events as indicators of community interest, my analysis of 2024 data reveals that ClickHouse stands out as the clear leader.

With approximately 6,400 new stars and a significantly higher number of forks than other projects, ClickHouse outperformed its competitors, more than doubling the attraction of Apache Doris, which ranks second.

Number of Repository forks in 2024 follow similar trend:

Apache Doris and StarRocks are having a close race to capture the attention of the industry and gain adoption as leading hybrid real-time OLAP and Data Warehouse engines.

In contrast, Apache Pinot and Apache Druid are struggling to keep pace with the likes of ClickHouse and Doris/StarRocks. The lack of significant innovations from these projects in 2024 might hinder their ability to further capture market interest.

Code Activity

Development activity in 2024 showed interesting patterns across projects. Apache StarRocks and Apache Doris led in pull request activity, each processing approximately 30K pull requests (opened plus closed), while ClickHouse maintained a strong third position.

However, ClickHouse demonstrated the highest code commitment activity, showing the most consistent push frequency and merge operations. StarRocks and Apache Doris followed with high activity levels, ranking second and third respectively.

In contrast, 2024 saw lower levels of development activity for Druid, Pinot, and Kylin. While Druid and Pinot are mature products that naturally might require less frequent updates, their reduced code activity, along with Kylin's, still raises concerns about potential project stagnation.

This trend is particularly particularly evident in Apache Kylin, which recorded only 150 pull requests and 85 code pushes for the entire year, indicating significantly diminished development momentum.

As a result, Apache Kylin will be excluded from the majority of this study moving forward.

User Engagement

Repository issue activity—both opened and resolved—serves as a critical indicator of project health and community engagement in open-source projects.

Analysis of 2024 data shows ClickHouse leading in this metric with the highest volume of user-reported issues and resolutions, with StarRocks and Doris showing strong activity levels in second and third positions respectively.

Project Collaboration

ClickHouse maintains the largest contributor base with over 1,600 committers since inception, yet Apache Doris and StarRocks demonstrated superior community growth in 2024, in attracting new collaborators.

The contribution metrics also reveals significant regional concentration, with StarRocks and Doris receiving substantial backing from major Chinese technology companies including Baidu, Tencent, and Alibaba. Over 80% of contributions to these repositories originate from China, reflecting strong regional investment in these platforms.

Adoption & Installations

Industry adoption and usage can be measured through download and installation metrics, with Docker Hub downloads serving as a particularly reliable indicator for database systems.

In this metric, ClickHouse demonstrates exceptional market penetration with over 100 million downloads.

Apache Pinot and Apache Druid show substantial adoption with 10 million and 5 million downloads respectively. The newer entrants, StarRocks and Doris, have achieved encouraging early traction with 500K and 100K downloads respectively.

Major 2024 Trends

1. Adoption of Decoupled Storage and Compute Architecture

Distributed real-time OLAP systems have traditionally followed the shared-nothing architecture common to modern MPP-based storage systems.

However, the industry's shift towards decoupled storage and compute models has prompted Real-time OLAP engines to embrace "zero-disk architecture".

This architecture which leverages deep storage solutions like HDFS and S3 as the primary persistence layer, offering enhanced scalability and flexibility while reducing operational costs.

In 2024, StarRocks and Apache Doris incorporated this architectural approach into their platforms.

ClickHouse had earlier laid the groundwork for this transition with its S3-backed MergeTree tables in version 21.8 (August 2021), enabling direct table storage in Amazon S3 or compatible object storage, and has since expanded its cloud offerings around this model.

2. Federated Analytics

The rising adoption of data lake and lakehouse architectures has prompted major analytical storage systems to pursue seamless integration with open table formats including Hudi, Iceberg, and Delta Lake.

Beyond these table formats, the platforms are extending their capabilities to support both read and write operations for industry-standard data lake file formats such as Parquet and ORC.

Modern real-time OLAP engines have embraced this trend, evolving beyond their traditional roles into unified analytics platforms.

ClickHouse, Apache Doris, and StarRocks have implemented native federation capabilities, enabling direct querying across diverse data sources—including data warehouses, data lakes, and open table formats—without the traditional requirements of data ingestion or replication.

3. Real-Time Data Warehouse

A significant trend in the ecosystem is the drive towards delivering "real-time data warehouse" capabilities.

This advancement enables ad-hoc analytical queries without pre-aggregation, supporting complex joins across multiple datasets—a functionality that has traditionally challenged real-time OLAP storage models.

Doris and StarRocks are leading this transformation by combining the strengths of flexible MPP engines (exemplified by Redshift, Snowflake, and BigQuery) with the real-time analytical capabilities of OLAP systems. Their hybrid approach achieves a balance of speed, flexibility, and scalability.

ClickHouse has also embraced this direction, enhancing its core engine to support broader OLAP workloads through improved update operations and enhanced join capabilities.

Major Features and Improvements Introduced in 2024

Below is an analysis of major features and enhancements introduced by each platform in 2024:

ClickHouse

• Refreshable Materialised Views: Enables periodic on-demand full recomputation and refresh of materialised views.

• Remote File Caching: Enhances efficiency for distributed workloads by caching remote files, significantly reducing access times.

• JSON Data Type: Introduced in ClickHouse v24.8, this new data type improves handling of semi-structured data.

StarRocks

• Shared-Data Clusters: Introduced in version 3.3, providing a decoupled storage and compute architecture for zero-disk capability using deep storage as the primary persistence layer.

• Pipe Service: Automates continuous loading of data files (e.g., Parquet) from deep storage services like S3 and HDFS into the StarRocks engine.

• Unified Catalog: Enables query federation over lakehouse tables, supporting direct queries on open table formats, with integration with Hive Metastore or AWS Glue for data discovery.

Apache Doris

• Decoupled Storage and Compute Architecture: Introduced in Doris 3.0.3 (December 2024), supporting S3-compatible object storage for cloud-native data persistence.

• Data Write-Back: Enables DDL and DML functions such as creating tables and writing data to Hive and Iceberg tables directly through Doris.

• Transaction Support: Adds CRUD operations like INSERT INTO SELECT, DELETE, and UPDATE.

• Materialised Views: Introduced asynchronous and multi-table materialised views.

• Semi-Structured Data Support: Enhanced with the new VARIANT data type.

• Expanded Data Lake Integration capabilities.

Apache Druid

• Version 30.0 Enhancements: Better ingestion experiences for Amazon Kinesis, Apache Kafka, Delta Lake, and improved integrations with Google Cloud Storage and Azure Blob Storage.

• Centralised Schema Management: Speeds up schema operations and cluster startup by gathering segment metadata.

• Version 31.0 (Dart Query Engine): Introduced a new query engine supporting complex workloads like large joins and high-cardinality GROUP BY, expanding Druid's capabilities into MPP (Massively Parallel Processing) territories.

Apache Pinot

• Multi-Stage Query Engine: Enhanced for better performance and scalability.

• Upsert and Compaction Improvements: Optimised for data ingestion workflows.

• Semi-Structured Data Support: Improved handling of JSON data.

• Delta Lake Integration: Added support for the Delta Kernel library, enabling integration with Delta Lake.

Major Vendor announcements and features

The following table lists key developments across SaaS vendors supporting these open-source platforms, including major announcements, strategic partnerships, and product expansions:

Features Comparison

The real-time OLAP systems has been extensively documented, with numerous comparisons of the top three contenders—ClickHouse, Druid, and Pinot.

Roman Leventov's comprehensive 2018 analysis stands as a notable reference point, detailing their architectural differences and capabilities. However, these platforms have evolved significantly in recent years.

The following sections provide a detailed comparison of these engines across key architectural and functional categories, reflecting their current capabilities and distinctions.

1. System Architecture

These platforms all build upon a distributed shared-nothing architecture. ClickHouse, Doris, and StarRocks have extended this model to support shared-storage (zero-disk) configurations through decoupled storage and compute capabilities.

From a software architecture perspective, ClickHouse, Doris, and StarRocks adopt a more simplified architecture compared to the relatively complex implementations of Apache Druid and Apache Pinot.

Doris and StarRocks are particularly notable for their self-contained design, operating independently of external system dependencies.

2. Data Architecture

All products implement columnar storage systems, with Doris and StarRocks recently extending their capabilities to support row-based storage modes.

Druid and Pinot distinguish themselves through their time-series-oriented data model, which leverages timestamp columns as primary partition fields. These two engines also excel in supporting hybrid workloads (real-time and batch) with flexible segmentation granularity.

Regarding transactional capabilities, Doris and StarRocks lead the ecosystem in ACID compliance, followed closely by ClickHouse. These platforms provide good support for primary key uniqueness, atomicity and concurrency control.

3. Query and Materialised Views

Materialised view support is currently limited to ClickHouse, Doris, and StarRocks.

These engines offer both synchronous materialised views which automatically update when source table data changes, and asynchronous materialised views which can be recomputed on demand or through scheduled jobs.

Doris and StarRocks extend this functionality with advanced features including multi-table materialised views and automatic query rewriting capabilities.

They also demonstrate superior complex join support, while ClickHouse provides moderate join capabilities. Druid and Pinot, however, are restricted to joins with small, dedicated dimension tables.

In terms of query processing capabilities, ClickHouse, Doris, and StarRocks maintain their leadership through sophisticated optimisation techniques, including cost-based optimization (CBO) and vectorised processing.

4. Data Ingestion

Real-time OLAP engines support two fundamental data ingestion modes: batch ingestion for processing data from sources like data lakes, and streaming ingestion for handling continuous data flows from platforms such as Apache Kafka.

These engines primarily employ ETL (Extract, Transform, Load) for data loading, performing transformations during ingestion to align with their immutable data model.

This approach particularly suits Druid, Pinot, and ClickHouse, which are optimised for denormalised and pre-aggregated data. The alternative ELT (Extract, Load, Transform) approach, which prioritises raw data loading, sees limited adoption in these systems.

For mutable data operations, Pinot, ClickHouse, Doris, and StarRocks provide support for upserts and primary-key-level row deduplication.

Using External Compute Frameworks

While offering native batch ingestion capabilities, these platforms integrate with external processing frameworks—including Hadoop (Druid, Pinot), Spark, and Apache Flink—to offload complex data transformation and computation during ingestion.

They also support push-based ingestion through frameworks like Kafka Connect and Flink CDC Connect, which enable data ingestion via custom-built connectors.

External storage support

All platforms effectively handle data lake file formats (CSV, ORC, Parquet), with ClickHouse, Doris, and StarRocks further extending support to major open table formats in data lakehouse architectures—a capability where Druid and Pinot currently lag.

In terms of log-based Change Data Capture (CDC) ingestion, ClickHouse and StarRocks offer comprehensive support, though ClickHouse's CDC solutions typically require subscription-based ingestion services.

5. Query Federation & Interfaces

As highlighted earlier, A key advancement in modern OLAP systems is their integration with external storage systems, enabling direct data querying without prior ingestion.

Among the five open-source platforms, ClickHouse, Apache Doris, and StarRocks lead with advanced query federation capabilities. These platforms support major external storage systems including data lakes, lakehouses, and DBMSs.

They enable external table definitions over files and directories in data lakes hosted on HDFS and S3-compatible platforms, and facilitate direct querying of open table formats like Hudi, Iceberg, and Delta Lake. Doris and StarRocks extend this support to include Apache Paimon.

Data Write-back

For data export and write-back operations, Doris and StarRocks support writing data to Hive and Iceberg, while ClickHouse mainly supports MySQL and Postgres.

Doris and StarRocks also enhance their data discovery capabilities through integration with external metadata services such as Hive Metastore and AWS Glue.

In terms of broader ecosystem integration, ClickHouse, Druid, and Pinot demonstrate comprehensive support. These major compute frameworks including Spark, Presto, Trino, and Hive, have integrated with these engines enabling direct data access.

All platforms provide moderate support for BI tool integration.

Recommendations

Each product in the current OLAP market offers unique advantages derived from its original design goals and subsequent feature developments.

While this general recommendation guide serves as a starting point, a comprehensive evaluation across multiple criteria is essential when comparing and selecting a product.

Small-to-medium Deployments:

Overall ClickHouse is an excellent real-time OLAP engine suitable for small-to-medium environments. Its straightforward deployment, management, and architecture make it the preferred choice for general use cases.

Large On-Premise Deployments:

For large-scale implementations, particularly on Hadoop or similar platforms, ClickHouse, Pinot, and Druid are leading candidates. The final selection should align with specific workload requirements and use cases.

Cloud-Native Implementations:

Cloud-native deployments utilising object storage as the main persistence layer can leverage managed solutions like ClickHouse Cloud, or platforms such as StarRocks and Doris. However, consider that StarRocks and Doris introduced their decoupled architecture recently, suggesting careful evaluation for production readiness.

Log Analytics & Time-series Data:

Druid and Pinot demonstrate particular strength in processing immutable time-series data, including web logs, machine logs, and clickstream events. Their support for hybrid tables makes them ideal for Lambda-style architectures.

Unified Analytics with Query Federation:

ClickHouse, StarRocks, and Doris excel in unified analytics scenarios, offering query federation capabilities that enable seamless data access across diverse sources such as data lakes, lakehouses and DBMS systems.

Hybrid Data Warehouse-OLAP Solutions:

StarRocks and Doris provide a middle ground, combining traditional data warehouse capabilities with real-time OLAP performance. They offer comprehensive CRUD operations, complex join support (including star schema), and ACID guarantees to some extend.

Conclusion

The real-time OLAP ecosystem has evolved from specialised engines for append-only data processing into versatile analytical platforms. While ClickHouse maintains its leadership position in general-purpose deployments, newer platforms like StarRocks and Apache Doris are bridging the gap between real-time analytics and data warehouse capabilities.

The adoption of decoupled architectures and unified analytics suggests continuing evolution in this space. Organisations should evaluate their specific requirements across performance, scalability, and integration needs while considering platform maturity and community support when making their selection from these open source systems.

Follow Me on LinkedIn

This is the third part in the Data Landscape Trends 2024-2025 series, focusing on the evolution of zero-disk architecture.

In the first part, we explored the evolution of the BI stack, while the second part examined the rise of single-node processing engines.

Introduction

The landscape of storage systems is continuously undergoing significant transformations. One major trend in this evolution is the emergence of cloud-native and "zero-disk" architectures for storage systems.

This shift represents a move away from traditional storage systems that rely on locally attached physical storage devices such as HDD, SSD or attached EBS volumes on cloud, towards designs that use remote, scalable object storage services as their primary persistence layer.

Understanding this transformation is crucial for data-intensive businesses as they navigate the future of data infrastructure, particularly as storage costs often account for a significant portion of total cloud infrastructure expenses.

In this article, I will explore the evolution of zero-disk architecture, studying:

• The historical context and limitations of traditional storage architectures.

• The emergence of disaggregated storage and its evolution in cloud environments.

• Economic drivers and technical trade-offs of zero-disk implementations.

• Current implementation patterns across different use cases.

• Future directions and emerging solutions.

Evolution of Storage Architecture

Traditional Approaches

Before the emergence of disaggregated storage systems, in addition to popular single-node disk-based database systems like MySQL and PostgreSQL, two dominant distributed storage architectures emerged: shared-nothing architecture and shared-disk architecture. Each approach presented unique advantages and challenges which we will explore further.

Shared-Disk Architecture

Early distributed storage systems were built with shared-disk architecture that leverages multiple nodes with independent compute resources (CPU and RAM) while sharing access to a common storage system connected via a network.

While shared-disk architecture provides a simplified data management, this architecture has significant challenges, such as I/O contention, which can become a critical bottleneck during large I/O-intensive ETL workloads due to the shared nature of the I/O subsystem across all nodes in the cluster.

Furthermore, shared-disk systems often necessitate specialised hardware and networking setups, such as Network Attached Storage (NAS) or Storage Area Networks (SAN), which add to the overall cost and complexity.

This design was frequently employed in traditional data warehouses, such as Oracle RAC, to distribute workloads efficiently across multiple servers.

Shared-Nothing Architecture

The shared-nothing architecture, notably outlined by Michael Stonebraker in a published paper in 1986, takes a different approach.

It distributes the system across multiple nodes, where each node operates independently with its own CPU, memory, and disk. Unlike shared-disk systems, nodes in a shared-nothing setup do not share resources, making this architecture highly scalable and efficient for distributed workloads.

One of the key advantages of shared-nothing architecture is its near-linear scalability. Systems can grow seamlessly by adding more nodes, enabling them to handle increased workloads effectively. Additionally, shared-nothing systems are cost-effective, as they rely on commodity hardware rather than expensive, specialised equipment.

However, shared-nothing systems are not without challenges. Managing data distribution is a key hurdle, as hotspots and data skews require careful partitioning and load balancing to ensure optimal performance. Similarly, auto-scaling can be complex and costly, often requiring disruptive operations such as repartitioning and leader-follower replication.

Despite these challenges, shared-nothing architectures have become the backbone of modern disk-based distributed systems like Apache Kafka, Apache Cassandra and DynamoDB. They are also widely used in clustered data warehouses like Teradata and the initial design of Amazon Redshift.

The Emergence of Disaggregated Storage

The journey towards zero-disk architectures and the concept of disaggregated storage and compute began to gain traction with the rise of Hadoop and its associated ecosystem in the mid 2000s.

Hadoop's Distributed File System (HDFS) provided a novel approach to data storage, using commodity hardware to create a highly scalable distributed storage layer with built-in redundancy and fault tolerance.

This architecture laid the foundation for systems that decouple distributed storage from compute, enabling scalability and flexibility. Distributed compute frameworks like MapReduce and Spark operate independently of storage units, processing data stored in HDFS via high-throughput networks.

This architectural pattern evolved into a formalised data architecture, giving rise to modern data lake and, more recently, data lakehouse systems. These systems featured fully decoupled storage and compute layers, contrasting with the tightly integrated approach of traditional monolithic database systems:

Early Zero-Disk Systems

A new class of storage systems, including HBase, Solr, Hive, and Ignite, emerged within the Hadoop ecosystem using this paradigm, leveraging HDFS as their primary storage abstraction.

This new paradigm presented a new database architecture pattern:

Furthermore, this approach acknowledges that storage and compute requirements often grow at different rates. In practice, storage needs tend to increase rapidly as organisations collect more data, while compute demands may remain relatively stable, especially if the nature of data analysis remains unchanged.

The Rise of Cloud Storage

Hadoop-based decoupled storage systems revolutionised data infrastructure by separating compute and storage layers. However, managing these systems introduced significant challenges due to the operational complexity of running and maintaining large Hadoop clusters in data centers.

Organisations needed to hire highly sought-after Hadoop experts, a resource that was scarce and costly. While large tech companies like Yahoo, LinkedIn, and Twitter could afford this, smaller organisations and non-tech companies often faced difficulties in scaling and operating Hadoop effectively.

In a petabyte-scale on-premise Hadoop platform, we have operational challenges as part of routine activities. These include managing regular disk failures and replacements, rebalancing data across nodes when new nodes are added or removed, and addressing performance issues caused by slow data nodes as disk utilisation approaches critical thresholds (typically around 80%). Additionally, maintaining and scaling the Name Node master services as workloads and data volumes grow is a critical task.

Emergence of Amazon S3 Storage

The emergence of Amazon S3 and the growing momentum of cloud adoption in 2010s provided a transformative alternative.

Cloud object storage services like Amazon S3 offered a simpler, more scalable, and cost-effective solution for building big data applications, gradually replacing Hadoop HDFS in many scenarios.

In addition to eliminating the operational complexities of managing and scaling HDFS, cloud storage offers several key advantages:

• High Availability: Cloud storage providers typically guarantee up to 99.99% availability, whereas on-premises HDFS systems often experience regular downtimes due to maintenance, crashes, and hardware or software upgrades.

• Elasticity: Cloud storage removes the need for advance capacity planning, which often results in inaccurate resource estimates, and eliminates upfront hardware procurement costs.

• Unlimited Scalability: Cloud storage provides virtually infinite scalability without the need for regular capacity planning and hardware procurement.

• Multi-Tier Storage: Cloud storage offers different storage classes with seamless data migration between them, enabling cost optimisation.

• Cost Efficiency: The total cost of ownership for cloud storage, such as Amazon S3, is generally lower than on-premises storage like HDFS, which incurs a 3x replication overhead.

• True Zero-Disk Architecture: Cloud storage supports running multiple permanent or ephemeral compute clusters on a shared storage infrastructure, enabling a truly zero-disk architecture.

The success of decoupled storage and compute architectures in Hadoop platforms and the benefits of the new cloud storage inspired industry to replicate the same architecture on cloud.

Instead of relying on HDFS, new systems would leverage S3 as the primary storage backend, paired with cloud compute for processing power. This innovation gave birth to what we now recognise as cloud-native architectures.

Netflix was among the first adopters to replace HDFS with Amazon S3 on its Hadoop clusters running on AWS in the early 2010s. They utilised S3 as the central data lake storage, while Hive and Spark served as the compute engine operating on Amazon EMR clusters.

The Rise of Cloud-Native Data Vendors

The launch of two major cloud data vendors, Databricks in 2013 and Snowflake in 2014, both embracing the decoupled compute and storage model with cost-effective cloud storage as the foundation, signaled further solidification of the new storage paradigm.

Snowflake, in particular, emerged as an early pioneer in implementing a fully decoupled, commercial cloud-native data warehouse. Their influential paper published in 2016, introduced key innovations that redefined cloud-based data warehousing, demonstrating how object storage systems like Amazon S3 could serve as the primary storage layer for durability and scalability.

The paper also introduced a novel multi-cluster shared data architecture, enabling diverse workloads—analytics, reporting, and ETL—to run as separate compute clusters on a single shared data platform.

This approach, known as "virtual warehouses" in Snowflake's terminology, proved highly influential, inspiring other vendors like Amazon Redshift to adopt similar design.

The Zero-Disk Paradigm

The rise of cloud-native big data processing, coupled with the maturation of cloud object storage as a universal storage backbone, has driven a shift towards a true 'zero-disk' architecture.

Unlike disk-based distributed systems that rely on shared-nothing architectures—requiring complex operations like load balancing and partition reassignment—zero-disk systems eliminate such overhead.

Before exploring the current landscape of zero-disk database systems further, let's examine the some of the key trade-offs in disk-based vs cloud-native zero-disk architecture.

The Economics of Zero-Disk Architecture

A key driver for adopting zero-disk architecture is the potential for substantial cost reduction.

From a storage cost perspective, local SSDs attached to VMs and even durable EBS volumes are significantly more expensive than cloud object storage. For instance, storing three replicas on attached storage can cost 10-20x more per GB compared to using cloud object storage.

Beyond the cost per GB, disk-based storage systems face additional expenses from cross-availability zone (AZ) data transfer fees when deployed in high availability mode with replication across multiple AZs. These inter-zone networking costs can significantly inflate operational budgets.

As Claimed by WarpStream, a cloud-native alternative to Kafka, and acknowledged by Redpanda and Confluent, infrastructure costs for running such systems in the cloud can account for 70% to 90% of total infrastructure and workload expenses. This is largely due to inter-zone networking overhead for replicating data between Availability Zones.

In contrast, distributed zero-disk systems like WarpStream leverage object storage (e.g., Amazon S3) as the backbone, claiming to reduce total cost of ownership (TCO) by 5-10x compared to Kafka.

For example, WarpStream reports that it can eliminate $641/day in inter-zone networking fees in a typical Kafka production deployment, replacing it with less than $40/day in object storage API costs. Similarly, Redpanda acknowledges the difficulty of escaping such hidden costs in traditional replicated systems.

Technical Trade-offs: Disk-Based vs. Zero-Disk

The fundamental trade-off in zero-disk architecture can be summarised as: "relaxing latency requirements to leverage cloud durability and economics of scale".

This trade-off manifests in several key areas:

1. Performance Characteristics

Traditional local disk-based storage offers fast access speeds (both for reads and writes) and high IOPS rate, but often lacks inherent scalability and redundancy features. To achieve durability and protection against data loss, disk-based systems require complex setups like RAID and software-level replication for fault tolerance.

Zero-disk storage systems, on the other hand, leverage the inherent redundancy and fault tolerance of object stores. Deep object storage systems provide safety guarantees, fault tolerance, and versioning out-of-the-box.

This simplifies system design, removing the requirement to manage data replication, backups, and snapshots to protect against data loss or disasters. However, the price for these benefits is the higher access latency, largely due to inherent object storage design and network I/O.

2. Directory and File listing Bottleneck

Listing operations for directories and files have been a significant performance bottleneck in distributed filesystems and cloud object stores. This challenge particularly affects large-scale data systems that need to manage extensive collections of files and partitions.

In query execution, the planning phase can sometimes exceed the actual processing time, especially when dealing with heavily partitioned datasets and large number of small data files.

Those who have worked with query engines like Hive operating on large HDFS cluster may have seen queries which require several minutes just to complete bulk listing operations, and gathering required metadata for all relevant files and partitions before processing can begin.

3. API call Limits

The challenge is further compounded by rate-limiting and throttling mechanisms implemented by cloud providers. Cloud object stores impose various operational limits that require careful consideration:

• Amazon S3, for example, imposes a limit of 1,000 objects per LIST request when using prefix listing functionality. To overcome this limitation, systems must execute hundreds of parallel calls, and use random prefixes, while each operation typically requiring between tens and hundreds of milliseconds to complete.

• S3 has a limit of 3,500 PUT/COPY/POST/DELETE or 5,500 GET/HEAD requests per second per prefix in a bucket. Therefore prefix management and distribution is important to get the level of horizontal scalability and parallelism required in large scale data applications when processing data on S3.

Mitigating Access Latencies

The higher latency associated with cloud object storage is a crucial challenge, particularly for real-time applications. Several techniques are currently used to mitigate this:

• External Metadata Management - To address the high latencies associated with listing files and directories, it is possible to explicitly maintain metadata records for files and partitions to alleviate this bottleneck. Open Table Formats like Apache Iceberg, Hudi, and Delta Lake store file and partition metadata in log files, minimising expensive API calls during the query planning phase by providing direct access to this information.

• Local Caching: Compute nodes can use fast local storage (e.g., NVMe SSDs) as a read-through and write-through cache. This reduces latency by storing hot data locally, and serving writes before flushing to object stores.

• Exploiting Throughput: Leveraging the high throughput of object stores by using multiple I/O threads can help improve access speed by executing many parallel I/O operations.

• Express Object Storage: New types of object storage, such as Amazon S3 Express One Zone, offering lower latency but with higher costs.

These approaches aim to balance the benefits of cloud object storage with the performance needs of various applications.

Implementation Patterns

Different use cases have led to distinct implementation patterns for zero-disk architecture, each addressing specific requirements and constraints.

Analytical Workloads on Deep Storage

As previously highlighted, analytical systems were among the pioneers in adopting zero-disk architecture, largely due to their ability to tolerate higher latencies.

Distributed Compute frameworks like Hive, Spark, and Presto/Trino have been processing petabyte-scale analytical workloads on cloud-based data lakes for over a decade.

The emergence of Snowflake and the adoption of a similar decoupled architecture by Redshift and BigQuery further advanced the implementation of OLAP database systems on deep storage. These platforms demonstrated how separating storage and compute could deliver scalability and flexibility without compromising performance.

To address challenges such as latency and API call overhead, several optimisation techniques are used including external metadata management to reduce expensive API calls, using efficient columnar storage formats like Parquet optimised for cloud object stores, and parallel processing to leverage high throughput capabilities to deliver high performance while benefiting from the cost efficiency and scalability of cloud-native architectures.

Building on the success of major vendors, the database industry has increasingly embraced this architectural shift, with numerous implementations emerging in recent years and 2024 in particular.

A recent example is the release of InfluxDB 3.0, announced in 2024, which represents a comprehensive architectural overhaul. This revamped system leverages Apache Arrow for in-memory buffering and Parquet as the storage format stored on remote cloud storage service.

Transactional Systems on Deep Storage

While zero-disk architecture has shown considerable success with analytical workloads and offline systems such as data warehouses, implementing transactional systems presents unique challenges.

OLTP database systems demand strict ACID properties and sub-second latency requirements, including the ability to perform fast in-point updates. These requirements create significant challenges when working with immutable storage layers with high I/O latencies that weren't designed for small random writes and updates.

Core Challenges

Several fundamental limitations of cloud object stores make implementing full ACID guarantees challenging:

Eventual Consistency

Prior to AWS S3's introduction of strong consistency in 2021, all INSERT, PUT, and DELETE operations were eventually consistent. This limitation could lead to inconsistencies where changes were not immediately visible due to the lack of strong read-after-write consistency—a critical requirement for transactional systems.

Lack of Mutual Exclusion

The lack of mutual exclusion capabilities in object stores presents another significant hurdle. Without built-in support for concurrent writes or "put-if-absent" guarantees, systems risk data loss or corruption from simultaneous writes to the same location.

Modern object storage frameworks like Apache Hudi and Delta Lake have addressed this by implementing external locking services, using systems like DynamoDB to manage concurrent writes to S3.

A significant development occurred in 2024 when AWS introduced "Conditional Writes" for S3, enhancing reliability and efficiency for data operations in S3 object storage. This feature ensures writes only occur if certain conditions are met, reducing the risk of unintentional data overwrites and improving concurrency control.

Non-Atomic File Operations

Traditional systems like HDFS handle file renaming as fast, atomic metadata operations. In contrast, cloud object stores must physically copy and delete data to achieve the same result. This limitation significantly impacted systems such as Hive, Spark, and HBase that rely on atomic rename operations for managing data files.

Given the constraints outlined above, is it feasible to design a general-purpose transactional database system, such as PostgreSQL, that utilises object storage as its primary storage layer? Let’s find out.

Hybrid Architecture Solution

While implementing fully zero-disk transactional systems remains challenging, a hybrid architecture has emerged as a practical solution.

This approach uses a fast replicated write cache combined with periodic offloading to deep storage, based on the LSM-Tree algorithm found in modern key-value stores like Cassandra and RocksDB.

The LSM-Tree architecture, originally designed to eliminate high I/O latencies from random I/O operations on HDDs, proves equally valuable for cloud storage systems.

This design helps manage both the high I/O latencies and lack of transactional primitives in cloud storage services. Key components include:

• A write-ahead log (WAL) maintained on local storage (such as EBS volumes) for atomic update guarantees.

• In-memory caching with periodic flushing to object storage to eliminate the high I/O cost of performing real-time updates.

• Query execution that reconciles both in-memory cache and object store data.

This architecture effectively addresses the high IOPS requirements of traditional database systems, which typically make numerous small changes at the page level.

Several modern systems have successfully implemented variations of this approach:

• Neon's Serverless PostgreSQL implementation uses this pattern in its cloud-native database.

• AutoMQ employs EBS volumes for write caching and WAL while continuously offloading data to remote storage.

• SlateDB embedded database implements an LSM-Tree architecture directly on cloud storage.

Future Directions

The general availability of S3 Express One Zone, announced in 2023, offering single-digit latency, suggests potential future developments for transactional workloads on deep storage. This service could potentially replace the replicated write cache layer presented in the hybrid architecture, enabling fully zero-disk transactional systems.

However, its current limitations—including costs seven times higher than standard S3 and single-zone replication—mean that hybrid approaches are likely to remain prevalent in the near term.

Real-Time Systems on Deep Storage

Similar to OLTP database systems, real-time event-based systems like Apache Kafka rely on fast I/O and certain ACID guarantees.

Constructing low-latency, real-time systems on deep storage presents unique challenges, as traditional disk-based systems depend on high IOPS and low-latency sequential I/O. Writing directly to cloud storage can introduce latencies in the hundreds of milliseconds, which is unacceptable for many real-time scenarios.

One solution is to mirror the approach presented for transactional systems: combining intermediate fast local write caches with periodic flushing to deep storage using LSM-Tree architecture.

AutoMQ implements this strategy by using EBS volumes for write-ahead logging while periodically offloading data to object stores. This architecture allows them to achieve single-digit millisecond P99 latency while still benefiting from cloud storage economics.

Full Zero-Disk Implementation

You might ask, is it possible to completely eliminate intermediate cache for real-time event-driven systems and go fully zero-disk?

Yes, but this requires accepting the primary trade-off: increased latency.

As WarpStream co-founder argues:

If you can tolerate a little extra latency, Zero Disk Architectures (ZDAs), with everything running directly through object storage with no intermediary disks, are better. Much better.

WarpStream represents a pioneering implementation of this approach. Built as a Kafka-compatible platform directly on cloud storage, it demonstrates both the possibilities and trade-offs of full cloud-native zero-disk architecture to take advantage of the cloud economics and eliminate exorbitant cost of running Kafka on cloud.

The founder asked:

“What would Kafka look like if it was redesigned from the ground up today to run in modern cloud environments, directly on top of object storage, with no local disks to manage, but still had to support the existing Kafka protocol?”

The system uses stateless "Agents" instead of traditional brokers, eliminating the need for JVM management and rebalancing operations. However, it accepts higher latencies—around 600ms for write confirmation and over one second for end-to-end operations—in exchange for significantly lower costs and operational simplicity.

For organisations willing to accept slightly higher latency, WarpStream offers a massively simplified and far more cost-effective, cloud-native alternative to Kafka.

This innovative design, combined with its Bring Your Own Cloud (BYOC) deployment model that fully decouples compute from storage, attracted Confluent, which acquired WarpStream in 2024 to integrate its groundbreaking architecture into Confluent's offerings, further expanding deployment options.

Industry Evolution

Beyond WarpStream, the real-time pub/sub industry has seen rapid evolution in this space:

Pinterest led early adoption, implementing an in-house Kafka alternative pub/sub system called MemQ in 2020 that proved to be 90% cheaper than traditional Kafka deployments with three-way replication across availability zones.

Redpanda made cloud storage their default storage tier in 2022, adopting a cloud-first approach. In 2024, they introduced Cloud Topics where data is written directly to cloud storage while only metadata is replicated across availability zones.

As highlighted earlier, AutoMQ launched in 2023 as a zero-disk Kafka alternative aims to replace disk with object storage as the persistence layer while maintaining the same performance and access latencies by using locally attached cache.

Confluent's trajectory shows the industry's direction. They announced Kora in 2023, a proprietary cloud-native engine, followed by Freight Clusters in 2024. These innovations enable direct writing to object storage, eliminating costly cross-AZ broker replication for workloads that can tolerate higher latency, achieving up to 90% cost reduction compared to traditional deployments.

Real-Time OLAP and Stream Processing

The trend extends beyond messaging systems. Apache Doris 3.X, released in late 2024, introduced compute-storage decoupled deployment mode for its OLAP engine, enabling the storage layer to operate on low-cost storage like HDFS and S3.

StarRocks OLAP engine v3.0, launched in 2023, introduced a shared-data cluster featuring a storage-compute separation architecture, enabling data persistence into S3-compatible object storage.

In 2024, StarRocks announced preview support for AWS Express One Zone as storage volume in version 3.3, to significantly enhance read and write performance on cloud storage.

ClickHouse Inc launched ClickHouse Cloud in 2022, with a separate storage and compute architecture, optimised for storing data on shared cloud object store , while using local NVMe SSDs for caching layer both for reads and write paths in order to mitigate the added access latency introduced by the object storage.

Apache Flink's 2.0 release in late 2024 marked another milestone, introducing a cloud-native architecture for state management. This replaced their previous complex tiered storage approach with a simpler model using deep storage as the primary storage layer and local disks as optional cache.

The Future of Real-Time Systems

The evolution of real-time systems on deep storage reveals a clear trend:

The success of these implementations suggests that the future of real-time systems might involve a spectrum of solutions, from hybrid approaches that prioritise performance to full zero-disk implementations that optimise for cost and simplicity.

The choice between these approaches will likely depend on specific use cases and latency requirements rather than technical limitations.

Other Emerging Patterns and Solutions

In addition to the growing adoption of zero-disk architecture, other emerging architectural patterns are leveraging cloud storage without fully transitioning to a zero-disk model.

Heterogenous Tiered Storage

Modern systems are implementing sophisticated tiered storage strategies to balance performance and cost.

Cheap deep storage provides a compelling choice for long-term storage of historical cold data or alternatively provide a read-only replica outside of the main storage system to external query processors.

Beyond established storage engines like Apache Druid and Pinot, which utilise external deep storage to offload older data segments, a growing number of disk-based storage systems have adopted similar approaches in recent years.

Redpanda introduced their Archival Storage subsystem in 2021, enabling automatic movement of data between tiers, which moves large amount of data between the brokers and the remote cloud storage automatically with the objective of providing infinite data retention with good performance at a low cost,

Apache Kafka has also embraced this approach. With the release of Kafka 3.6 in 2023 initially proposed by Uber, the platform introduced tiered storage support with pluggable remote storage options. The system allows for configurable retention policies per tier and automates segment migration between storage layers.

Crunchy Data implemented a feature in their managed PostgreSQL service to copy data into the data lake for cheaper and long term storage supporting Parquet format.

Remote Read Replicas

Redpanda's Remote Read Replica functionality, introduced in 2022, represents another innovative approach to leveraging object storage in a semi-disk-based approach.

This feature enables separate consumer clusters to access archived data without impacting primary production clusters. It also allows for the integration of topics from multiple production clusters, providing unified read-only access for downstream consumers.

While the implementation of tiered storage and remote read replica by disk-based software systems such as Kafka and Redpanda try to take advantage of cloud-storage economics without a full re-architecture of their systems, however it still doesn't live up to the promise of reduced total cost and simplicity offered by a full zero-disk architecture.

Conclusion

The rise of zero-disk architecture represents a fundamental shift in storage system design, driven by cloud economics and the maturation of distributed object storage. While this approach introduces certain trade-offs, particularly around latency, the benefits in terms of cost reduction, operational simplicity, and built-in reliability make it an increasingly attractive option for modern applications.

As cloud infrastructure continues to evolve and new techniques for managing latency emerge, the balance between performance and cost will likely continue to shift in favour of zero-disk architectures, particularly for organisations prioritising cost optimisation over sub-second latency requirements.

Recent industry developments further demonstrate the momentum behind zero-disk architectures. The introduction of cloud-native Kafka solutions like Confluent's Kora and Freight Clusters, which integrate directly with object stores in 2024, along with Confluent's acquisition of WarpStream, the launch of Flink 2.0 with a cloud-native architecture, The full re-architecture of InfluxDB, and Redpanda's emphasis on cloud-first strategies and cloud topics, all signal a growing shift toward cloud-native, zero-disk architectures as a dominant paradigm for modern storage systems.

Follow Me on LinkedIn

This is part two of Data Landscape Trends 2024-2025 series, focusing on single-node processing trends.

Introduction

2024 witnessed growing interest in single-node processing frameworks, with tools like DuckDB, Apache DataFusion, and Polars receiving increased attention and gaining unprecedented popularity from the data community.

This trend represents more than just a technological advancement—it marks a fundamental reassessment of how we approach data analytics.

As we move away from the "big data" era's distributed-first mindset, many businesses are discovering that single-node processing solutions often provide a more efficient, cost-effective, and manageable approach to their analytical needs when their size of data is not that big.

When I recently published a short post on LinkedIn titled "Why Single-Node Engines Are Gaining Ground in Data Processing", I didn’t anticipate the significant attention it would receive from the LinkedIn data community. This response underscored the industry’s increasing interest in the topic.

In this article, I will dive deeper into the subject, exploring it in greater detail and providing further insights.

Rethinking Big Data

The past decade saw many businesses scrambling to implement big data strategies, with many companies investing heavily in distributed processing frameworks like Hadoop and Spark.

However, recent analyses reveal a surprising truth: most companies don't actually have "big data".

A significant majority of companies do not require large data platforms to address their data analytics needs. Often, these companies are swayed by marketing hype and make substantial investments in these platforms, which may not effectively resolve their actual data challenges.

Jordan Tigani, a founding engineer on Google BigQuery, analysed usage patterns and found that the median data storage size among heavy BigQuery users is less than 100 GB.

Even more revealing, an analysis of half a billion queries run on Amazon Redshift published in a paper showed that:

• Over 99% of queries processed less than 10 TB of data.

• Over 90% of sessions processed less than 1 TB.

The paper also states that:

Most tables have less than a million rows and the vast majority (98 %) has less than a billion rows. Much of this data is small enough such that it can be cached or replicated.

This analysis reveals that with a big data processing threshold of 1 TB, over 90% of queries fall below this threshold.

As a result, single-node processing engines have the potential to handle workloads that previously required distributed systems like Spark, Trino, or Amazon Athena to process across multiple machines.

This reality challenges the common notion that big data infrastructure is a necessity for all modern businesses.

Workload Patterns & Rapid Data Aging

The case for single-node processing becomes even more compelling when we examine how organisations actually use their data.

Two key patterns emerge: the data aging effect and the 90/10 rule of analytical workloads.

The Data Aging Effect

As data ages, access frequency declines sharply. For the majority of companies, data access patterns follow a predictable lifecycle:

• Hot data (0-48 hours): primarily from ETL pipelines.

• Warm data (2-30 days): Accounts for most analytical queries.

• Cold data (30+ days): Rarely accessed but often retained for compliance or historical analysis.

A study of Meta and eBay's data access patterns revealed this sharp decline in access after the first few days, with data typically becoming cold after a month.

In our analysis of a petabyte-scale data lake, we found that raw data remains hot for only 48 hours, with 95% of access occurring in that time, mainly by downstream ETL pipelines. In Analytics (Gold) zone, the hot period lasts about 7 days, and 95% of queries are executed only within 30 days.

The 90/10 Rule for Analytical Workloads

This aging effect leads to the 90/10 rule in analytical workloads:

If the combined hot and warm period is 30 days accounting for 90% of workloads, then, with a one-year retention period, over 90% of workloads access fewer than 10% of the data.

This pattern holds remarkably consistent across industries and use cases. Even in organisations with large datasets, most analytical workloads operate on recent, aggregated data that could easily fit within single-node processing capabilities.

Hardware Evolution & Rethinking Scale Up

The capability of single-node systems has grown exponentially since the early days of big data.

The rationale and motivation behind the scale-out strategy which became popular with emergence of Hadoop in mid 2000s in data processing is the necessity of combining multiple machines to address scaling challenges, enabling efficient processing of large datasets within reasonable timeframes and performance levels.

By integrating multiple machines in distributed systems, we effectively create a single large unit, pooling resources such as RAM, CPU, disk space, and bandwidth into one large virtual machine.

However, we need to reassess our assumptions about distributed processing and the scaling challenges faced in the 2000s to see if they remain valid today.

In 2006, when Hadoop MapReduce emerged, the first AWS EC2 instances (m1.small) had just 1 CPU and less than 2 GB RAM. Today's cloud providers offer instances with 64+ cores and 256GB+ of RAM, fundamentally changing the equation for what's possible with single-node processing.

Examining the evolution of balanced EC2 instances in terms of memory and CPU (with a 1:4 ratio) over the years reveals exponential growth, as these instances have become increasingly powerful over time.

The Economics of Scale-Up vs. Scale-Out

One might assume that scaling out across multiple smaller instances is more cost-effective than using larger instances. However, cloud pricing models tell a different story.

The cost per compute unit on cloud is consistent whether you use a smaller instance or larger one as the cost increases linearly.

That is the cost of larger cloud compute instances increases linearly and the overall price remains the same regardless of whether you use one larger instance or multiple smaller instances, as long as the total number of cores and memory is the same.

Using AWS's m5 instance family as an example, regardless of whether you scale up with a single m5.16xlarge instance or scale out with eight m5.2xlarge instances, the price per hour will remain the same.

This hardware evolution has important implications for system architecture decisions as:

This raises a critical question:

From a cost-performance perspective, if a single-node query engine can handle the majority of workloads efficiently, is there still a benefit to distributing processing across multiple nodes?

The Performance Case for Single-Node Processing

Modern single-node processing engines leverage advanced techniques to deliver impressive performance.

Engines like DuckDB and Apache DataFusion achieve superior performance through sophisticated optimization techniques, including vectorized execution, parallel processing, and efficient memory management

Numerous benchmarks illustrate these performance improvements:

• Vantage reported that when they switched from Postgres to DuckDB for cloud cost analysis, they saw performance improvements between 4X and 200X.

• Fivetran's CEO benchmarks using TPC-DS datasets showed DuckDB outperforming commercial data warehouses for datasets under 300 GB.

• An experiment with 1 billion row fake order data, comparing DuckDB with Amazon Athena.

Why Choose Single-Node Processing?

The case for single-node processing extends beyond just performance. For the majority of businesses, modern single-node engines offer several compelling advantages:

• They dramatically simplify system architecture by eliminating the complexity of distributed systems. This simplification reduces operational overhead, makes debugging easier, and lowers the barrier to entry for teams working with data.

• They often provide better resource utilisation. Without the overhead of network communication and distributed coordination, more computing power can be dedicated to actual data processing. This efficiency translates directly to cost savings and improved performance.

• They offer excellent integration with modern data workflows. Engines like chDB and DuckDB can directly query data from cloud storage, work seamlessly with popular programming languages, and fit naturally into existing data pipelines.

• The embeddable nature of some of these engines enables seamless integration with existing systems—from PostgreSQL extensions like pg_analytics and pg_duckdb to various modern Business Intelligence tools—expanding analytical capabilities without disrupting established workflows.

Challenges and Limitations

While single-node processing offers many advantages, it's important to acknowledge its limitations.

Some engines still face challenges in fully utilising all available CPU cores on large machines, particularly as core counts continue to increase. Memory hierarchy bandwidth between RAM and CPU can become a bottleneck for certain workloads.

When reading from cloud storage like S3, single-connection transfer speeds may be limited, though this can often be mitigated through parallel connections and intelligent caching strategies. And naturally, there remain workloads involving very large datasets that exceed available memory and storage, requiring distributed processing.

Conclusion

The rise of single-node processing engines represents a pragmatic shift in data analytics. As hardware capabilities continue to advance and single-node engines become more sophisticated, the need for distributed processing will likely continue to decrease for most organisations.

For the vast majority of companies, single-node processing frameworks offer a more efficient, cost-effective, and manageable solution to their data analytics needs. As we move forward, the key is not to automatically reach for distributed solutions, but to carefully evaluate actual workload requirements and choose the right tool for the job.

The future of data processing may well be less about managing clusters and more about leveraging the impressive capabilities of modern single-node systems.

Follow Me on LinkedIn

As we dive into 2025, the data engineering field continues its dramatic evolution. In this series, we'll explore the transformative trends reshaping the data engineering landscape, from emerging architectural patterns to new tooling approaches.

This is part one of our series, focusing on the evolution of Business Intelligence architecture.

Introduction

The Business Intelligence (BI) landscape has undergone significant transformation in recent years, particularly in how data is presented and processed.

This evolution reflects a broader shift from monolithic architectures to more flexible, composable solutions that better serve modern analytics needs.

This article traces the evolution of BI architecture through several key phases: from traditional monolithic systems, through the emergence of headless and bottomless BI, to the latest developments in BI-as-Code and embedded analytics.

Traditional BI Architecture: The Monolithic Approach

Traditional BI tools were built as comprehensive, tightly-coupled systems with a significant focus on user interface design.

These systems provided extensive flexibility through click-through functionality for slicing, dicing, and grouping data using various visualisations. At their core, these systems were composed of three interconnected components that worked in harmony to deliver business insights.

The backend layer served as the foundation, handling data ingestion from OLAP sources and building optimised data cubes on the server. These cubes contained pre-computed dimensions that enabled real-time data exploration.

Working in concert with the backend, the frontend layer provided the visualisation interface, connecting to the backend to access data cubes and construct dashboards.

The semantic layer completed the architecture by defining key performance indicators (KPIs) and metrics embedded within the BI software.

The Drawbacks of Traditional BI Tools

While powerful, these traditional systems came with significant overhead.

Organisations needed substantial infrastructure for on-premise deployment before managed Cloud BI services become more accessible, and the licensing costs were often prohibitive.

Implementation timelines stretched long, with even proof-of-concept demonstrations requiring weeks of setup and configuration. For businesses serving large user bases, the resource requirements were particularly demanding.

These fundamental limitations, combined with the growing need for flexibility and cost-effectiveness, sparked a series of architectural innovations in the BI landscape.

The Rise of Bottomless BI Tools

In response to these challenges, a new generation of lightweight, disaggregated BI tools emerged. Notable open-source solutions like Apache Superset, Metabase, and Redash began appearing about a decade ago, with Superset, originally developed at Airbnb, gaining particular prominence in the ecosystem.

These new tools adopted a "bottomless" architecture, eliminating the heavy backend server traditionally used for computation and building and caching cube objects.

Instead of maintaining their own computation layer, they rely on connected source engines for querying and providing data to dashboards at runtime. This architectural shift introduces different strategies for data serving.

Dealing With Query Latency

The elimination of the backend report server presents bottomless BI tools with a significant challenge: managing query latency when accessing data in real-time.

To address this challenge, these tools employ several optimisation strategies. One key approach involves utilising pre-computed aggregates stored in the primary data warehouse, allowing dashboards to serve results efficiently.

Additionally, tools like Superset implement caching layers using Redis to store frequently accessed datasets. This caching mechanism proves particularly effective: once an initial query loads a dataset, subsequent visualisations and dashboard reloads can access the cached version until the underlying data changes, significantly reducing response times.

For companies handling larger data volumes, integration with specialised real-time OLAP engines like Druid and ClickHouse provides low-latency analytics capabilities.

The Emergence of Universal Semantic Layer

As the industry sought more flexibility in their BI stack, portable semantic layer or what is known as headless BI emerged as an intermediate step between traditional monolithic systems and fully lightweight solutions.

Headless BI platforms provide a dedicated semantic layer and some combine query engine while allowing organisations to use any frontend visualisation tool of their choice. This approach fully disaggregates the presentation layer (front-end) from the semantic layer.

With tools like Cube and MetricFlow (now part of dbt Labs), for example, organisations can define their metrics and data models in a central location, then connect various visualisation tools, custom applications, or lightweight BI solutions to this semantic layer.

This architectural pattern offers several advantages over traditional BI systems. It enables organisations to maintain consistent metric definitions across different visualisation tools, supports multiple frontend applications simultaneously, and provides better integration capabilities with modern data stacks.

The semantic layer acts as a universal translator between data sources and visualisation layers, ensuring consistent business logic across all analytics applications.

The BI-as-Code Movement

Recent years have witnessed the emergence of BI-as-Code, representing an even lighter approach to dashboard and interactive data app development.

This paradigm shift brings software engineering workflows to BI development, enabling version control, testing, and continuous integration practices. Since code serves as the primary abstraction rather than a user interface, developers can implement proper development workflows before deploying to production.

Prominent tools in this space, such as Streamlit, integrate seamlessly with the Python ecosystem, allowing developers to remain within their Python projects without requiring external software installation for building dashboards and data applications.

The approach emphasises simplicity and speed, using SQL and declarative tools like YAML for dashboard creation. The resulting web apps can be easily self-hosted, providing deployment flexibility.

While Streamlit leads the pack in popularity, open source newcomers like Evidence, Rill, Vizro, and Quary have emerged in recent years, each bringing their own approach to the BI-as-Code concept.

BI-as-Code Limitations

BI-as-Code tools currently have limitations in terms of interactive data exploration features and providing enterprise-grade BI capabilities.

They don't provide the same user experience for slicing and dicing data as traditional BI tools, and they lack data governance and semantic layer support found in both traditional and lightweight BI solutions.

Nevertheless, BI-as-Code are increasingly being adopted in various ways such as data science teams creating interactive standalone apps, product teams building embedded analytics features and analysts developing internal data applications.

The New Emerging Trend: BI + Embedded Analytics

The latest evolution in BI architecture involves integrating high-performance, embeddable OLAP query engines like Apache DataFusion and DuckDB.

This approach bridges several gaps in the current landscape while maintaining the benefits of lightweight, disaggregated architectures.

The new full-stack composable BI architecture brings several key advantages:

• First, it offers true composability and interoperability, with ability to swap embedded compute engines as needed while maintaining a standalone semantic layer for metric definitions.

• The embedded analytics capabilities are particularly powerful, with zero-copy integration through standard frameworks mainly Apache Arrow enabling microsecond-level data access through optimised in-memory columnar formats.

• The direct support for data lakes and lakehouses represents another significant advance, allowing teams to build dashboards directly on top of open table formats like Apache Iceberg and Apache Hudi without intermediate data movement.

• This capability, combined with comprehensive federated query support, resolves a long-standing challenge in exisitng lightweight BI tools that struggled to effectively combine data from multiple sources without requiring to use an external federated query engine.

Industry Adoption

Industry adoption of embedded query engines is gaining significant momentum across the BI ecosystem. Commercial vendors are leading this transformation: Omni has integrated DuckDB as its core analytics engine, while Cube.dev has implemented a sophisticated combination of Apache Arrow and DataFusion in its headless BI architecture.

Similarly, GoodData has embraced this trend by implementing Apache Arrow as the foundation of its FlexQuery system's caching layer, and Preset (Managed Superset) has integrated with MotherDuck (Managed DuckDB ).

In the open-source space, both Superset (using duckdb-engine library ) and Metabase now support embedded DuckDB connection, with potential future integration into their core engines.

The BI-as-Code movement has also embraced embedded engines. Rilldata announced DuckDB integration in 2023 for auto-profiling and interactive modeling in dashboard development, while Evidence introduced Universal SQL in 2024, powered by DuckDB's WebAssembly implementation.

Conclusion

The Business Intelligence landscape continues its evolution toward more flexible, efficient solutions.

Each architectural evolution has brought distinct advantages: headless BI enabled consistent metrics across tools, bottomless BI reduced infrastructure complexity, BI-as-Code brought developer workflows to analytics, and embedded engines are now combining these benefits with high-performance query capabilities.

The integration of embedded query engines with lightweight BI tools represents a promising direction for implementation of lightweight BI, combining the best aspects of traditional BI capabilities with modern architectural patterns. As these technologies mature and the ecosystem grows, companies can look forward to increasingly sophisticated yet composable solutions for their data analysis needs.

Follow Me on LinkedIn

✨ Featured - The Rise of Single-Node Engines

In a recent blog post, Jordan Tigani, MotherDuck co-founder and former tech lead at Google BigQuery, highlighted that most companies don't actually deal with "Big Data." An analysis of half a billion sample queries run on Amazon Redshift revealed that over 80% of the queries processed less than 1 TB of data.

Even among the small percentage of businesses that do handle Big Data, the majority of queries (95%) are executed on smaller tables. In cases where companies have large datasets, with some tables exceeding 10 TB, about 96% of the queries still target smaller, likely aggregated tables of 100 GB or less, rather than the actual large tables.

A published paper by AWS also notes that most tables contain fewer than a million rows, with the vast majority (98%) having fewer than a billion rows.

These findings, combined with advancements in software and hardware technology that have significantly enhanced the processing capabilities of single-node systems, suggest that powerful emerging single-node compute engines like DuckDB could increasingly handle many non-big data use-cases. This would reduce the need for distributed processing frameworks like Spark, provided that the maturity and ecosystem support of these engines continue to evolve.

In a recent LinkedIn post, I shared a graph with some of these points, which sparked an engaging discussion among the data engineering community. The conversation featured a mix of opinions on the future of computing, and it's worth checking out.

💡Opinion

👉 The Infamous Rise of Notebook Engineers!

Daniel Beach has penned an intriguing article critiquing the growing use of notebooks in data engineering. He argues that many engineers misuse notebooks due to either a lack of technical skills or encouragement from vendors, like Databricks, promoting questionable practices. While notebooks are valuable for data analysts and scientists engaged in iterative data analysis, Daniel contends they are ill-suited for robust data engineering lifecycles. This misuse leads to poor coding standards, insufficient testing, and inadequate deployment practices. —> Read More

👉 The Analytics Personas in Business

Tristan Handy, the founder and CEO of dbt Labs, wrote an insightful piece about the key analytics personas in business. He critiques the common approach of treating the analytical process like an assembly line, which often fails to deliver significant insights and ROI. Handy suggests that the personas—primarily engineers, analysts, and decision-makers—should be seen as interchangeable "hats" that team members can wear when needed, while still maintaining their primary roles. This flexible approach fosters collaboration and enhances the overall effectiveness of the analytics process. —> Read More

📡 Open Source News

👉 Apache Airflow 2.10 Release

Apache Airflow 2.10 was released recently, introducing exciting new features such as support for multiple executors within a single Airflow environment. This allows users to assign different executors, like LocalExecutor and CeleryExecutor, to individual DAGs and even specific tasks. There are also numerous enhancements to Datasets and the UI, which are worth exploring. —> Read More

👉 Development of a New PostgreSQL DuckDB Extension

MotherDuck announced pg_duckdb, an open-source Postgres extension that embeds the DuckDB engine into the Postgres database for running analytical queries on Postgres data. This is a significant step towards easily transforming a popular OLTP system into an HTAP system using an embedded OLAP engine. The development looks promising, with multiple companies such as Microsoft, Neon, and Hydra joining the effort. The beta version has been released recently. —> Read More

👉 Ibis's Default Backend Change

The Ibis DataFrame library project announced that it will drop the Pandas and Dask backends in favour of making DuckDB its default backend. This decision is due to DuckDB's ease of installation, impressive speed, and strong support within the Python ecosystem. —> Read More

🛠 Practical Data Engineering

👉 Apache Airflow Alerting Techniques

The Google Data Analytics blog has provided a comprehensive overview of the alerting hierarchy in Apache Airflow, ranging from the top DAG level down to the individual task instance level. It details various alerting mechanisms that can be used to monitor the state of DAG runs and receive notifications about potential failures. The alerting techniques discussed are applicable not only to Google's Cloud Composer managed Airflow service but also to other Airflow deployments. —> Read More

👉 Best Practices for Optimising Airflow

The AWS blog has covered comprehensive strategies for optimising cost and performance in its Apache Airflow managed service, Amazon MWAA. Right-sizing remains crucial for achieving a balanced price-performance ratio in managed services. Amazon MWAA also supports auto-scaling, which can aid in this optimisation. The blog offers additional useful techniques for optimising DAG code to ensure that DAGs remain healthy, efficient, and scalable. These techniques can be applied to any Airflow deployment setup. —> Read More

Speaking of Airflow…

⚙️ Technical Deep Dive

👉 History and Evolution of Block Storage Services at AWS

Another fascinating story on the All Things Distributed blog explores the evolution of block storage services offered by Amazon Web Services (AWS). Written by one of AWS's leading engineers, it highlights key milestones in the development of Elastic Block Store (EBS), showcasing enhancements in performance, scalability, and continuous innovation. —> Read More

👉 The Internals of Apache Parquet

Vu has authored an insightful article on the internals of Parquet, the most popular cloud file format for data lakes. For those working with data lakes, understanding the design and architecture of common serialisation formats is invaluable for optimising storage and queries. —> Read More

👉 The Future of Distributed Systems and Their Storage Backend

A great article by Colin Breck on the future of distributed systems and highlighting current challenges, and major trends such as acceleration of object store adoption as the main storage backend abstraction for many analytical and transactional database systems. —> Read More

💪 Skill Up

In a recent LinkedIn post, I shared my top book recommendations for learning data engineering fundamentals. The feedback was overwhelmingly positive, with many comments emphasising the importance of selecting a good book and focusing on mastering the fundamentals. I have personally read all these books in recent years and have gained a lot from them.

👉 DataCamp’s Free Week

DataCamp is offering free access to its entire platform and all courses for a week, from November 4 to 10. This is a great opportunity to explore their courses and enhance your skills in the coming week! —> Read More

🔎 Case Studies

👉 Cost-effective Data Analytics Using Deterministic Sampling

Meta has shared valuable insights into its approach for achieving cost-effective data analytics through the use of deterministic sampling. This strategy is designed to balance the cost versus value trade-off, especially as data volumes and computation costs continue to rise exponentially. By employing deterministic sampling, Meta aims to reduce the overall cost and complexity of analytics without compromising the quality of insights. —> Read More

👉 Uber's New Declarative Batch ETL Framework

Uber has developed a modular declarative batch ETL framework called Sparkle, which leverages Apache Spark as the compute engine. Sparkle simplifies and standardises ETL pipeline development by allowing users to focus on expressing business logic as a sequence of transformation modules in SQL or Java/Scala/Python. It includes embedded unit testing and marks Uber's transition of all its batch ETL pipelines from Hive to Spark in 2023. —> Read more

👉 Self-service Kafka Platform Development Journey

Doordash has shared their journey in developing a self-service Kafka platform, aimed at addressing the challenges of managing Kafka infrastructure efficiently. This initiative was driven by the need to simplify the management of Kafka topics and resources, which was previously hindered by the use of low-level configuration management tools like Terraform. —> Read more

📣 Vendors News & Announcements

👉 Continuous Queries on Data Warehouse Systems

Google BigQuery has introduced a significant new feature called BigQuery continuous queries, currently available in Preview. This feature transforms BigQuery from a batch system into an event-driven streaming pipeline, leveraging the concept of Stream-Table Duality. It allows for the continuous ingestion of new events as data is loaded into BigQuery, enabling use cases like event-driven data processing, continuous record replication to a pub/sub queue or other streaming storage systems, real-time ML model integration, and Reverse ETL use cases. —> Read More

The Confluent blog has also published an article on leveraging this feature to stream data from BigQuery into the Confluent platform.

👉 New Google Managed Apache Kafka Service

Google has also announced a new managed service, Google Cloud Managed Service for Apache Kafka. This service abstracts the complexities of deploying and managing a Kafka cluster, offering features like security management, full management of brokers and storage, and automatic horizontal and vertical scaling. It also includes automatic storage tiering and lifecycle management to offload cold data to unlimited cloud storage. —> Read More

👉 Introduction of Conditional Writes on AWS S3

AWS recently announced the introduction of "Conditional Writes" on S3, marking a significant advancement in enhancing the reliability and efficiency of data operations, especially for distributed applications. This feature ensures that writes occur only if certain conditions are met, reducing the risk of unintentional data overwrites. It allows multiple clients to read and write to the same object without conflicts or concerns about overwriting each other's data. —> Read More

🎥 Conferences & Events

👉 Thinking Like an Architect

Gregor Hohpe delivered an insightful presentation titled "Thinking Like an Architect" at QCon London 2024. If you're interested in or work with data architecture, his talk provides valuable insights worth exploring. —> Watch

👉 Carnegie Mellon University's Intro to Database Systems Course - Fall 2024

The Fall 2024 session of Carnegie Mellon University's renowned "Intro to Database Systems" course was commenced in August. You can follow along with the course through recorded lectures available on their YouTube channel.

In my previous article, DuckDB Beyond the Hype, I explored its various use cases and briefly demonstrated how it can be used in data engineering and data science workflows.

One use case that particularly resonated with readers was using DuckDB for data transformation and serialisation on data lakes. Inspired by some readers feedback, I decided to write this follow-up article to dive deeper into this use case and provide a full code example.

In this article, I present a high-level use case with sample code, demonstrating how to move data between different zones in a data lake, using DuckDB as the compute engine.

To keep the focus on the core concepts, only brief code snippets are included, but the full implementation can be found on GitHub for those who want to dive deeper.

Project Overview

For simplicity, I implemented the project as a pure Python application, with minimal dependencies. The only external dependency is a cloud object store for our data lake implementation, which doesn’t have to be AWS S3 specifically—it can be any S3-compatible object store. Alternatively, you can use LocalStack to emulate the data lake locally on your machine.

The use case we’ll be exploring involves incrementally collecting the Github Archive datasets, which provide a full record of GitHub activities for public repositories, and enabling analytics on top of that data.

Data Lake Architecture

We will a multi-tier architecture, also referred to as Medallion architecture for this exercise. The Medallion architecture is a data lake design pattern that organises data into three zones:

• Bronze Zone: Containing raw, unprocessed data ingested from various sources.

• Silver Zone: Containing cleaned, conformed and potentially modeled data.

• Gold Zone: Containing aggregated and curated data ready for reporting, dashboards, and advanced analytics.

By maintaining this multi-zone architecture (Bronze → Silver → Gold), we ensure access to data at various stages of processing—ranging from raw data for detailed analysis to aggregated data for quick insights. This flexibility allows us to meet a wide range of analytical needs while optimising for both storage and query performance.

Partitioning Scheme

Each zone will use an appropriate partitioning scheme to optimise data ingestion, query performance and improve overall efficiency.

Typically, the partitioning scheme aligns with the data ingestion and transformation frequency for batch workloads. This approach ensures that the pipeline remains deterministic at the partition level.

In other words, there are no overlaps between individual job runs, and if a pipeline fails or an anomaly is detected in the data, we can safely rerun the job for that specific period. The pipeline will replace the affected partition and its contents, avoiding any negative side effects, data duplication or inconsistencies.

This approach is often referred to as functional data processing using immutable partitions. It was popularised by Maxim Beauchemen, who introduced this powerful data engineering pattern in a highly regarded blog post back in 2018.

Following the functional data processing pattern, we’ll ingest data hourly from the source and partition it by both day and hour in the Bronze Zone. Each hourly job writes atomically to its designated partition, ensuring data integrity at this level of granularity.

We’ll apply the same partitioning strategy in the Silver Zone, since the transformation from Bronze to Silver will also run on an hourly basis.

In the Gold Zone, partitioning will be done only by day, since the aggregation job runs on daily internal. The daily job will produce results for the previous day's data from the Silver Zone.

This approach strikes the right balance between performance and storage—using fine-grained partitions in the early stages and higher-level aggregation in the final layer.

The overall partitioning scheme for the three zones is illustrated in the figure below.

Data Pipeline Architecture

The end-to-end data pipeline is broken down into three key steps:

Step #1 - Ingest the hourly GitHub Archive batch dataset from the gharchive.org website over HTTP and load it into the Bronze Zone of our data lake.

Step #2 - Run an hourly transformation pipeline to clean and serialise the JSON files which loads the results into the Silver Zone.

Step #3 - Run a daily transformation pipeline to aggregate the data from the previous day and store it in the Gold Zone.

Data Source Exploration

Before we jump into building our data pipeline, we should explore and learn about the data source, its characteristics and the shape of data.

A great way for data exploration is using Jupyter. DuckDB can help with exploring the remote data without having to download the entire dataset locally.

The source data are available from gharchive.org and the data dumps are made available on different intervals such as hourly and daily.

The following example demonstrates how to analyse a sample gharchive dump file using DuckDB. With DuckDB, you can define a virtual table directly over a URL, enabling you to query and analyse the data without the need to download the file locally first.

Data Ingestion - 🥉 Bronze Zone

To implement our data ingestion pipeline, we need to identify the interface, protocol and data type of the source system, as this will guide our approach. Here's the breakdown for this use case:

Based on the above requirements, we need to download hourly JSON compressed files from gharchive.org server over HTTP.

A straightforward approach in Python is to use the requests library to stream and buffer the file from the source, followed by the boto3 library to upload the file to S3, publishing it to the Bronze zone of the data lake.

The Following is a simple example to collect and publish the data:

Data upload to S3:

While this example is suitable for testing, it's a basic implementation. To build a robust data pipeline we need to incorporate better parametrisation, error handling, and modularity.

For this, I’ve written a class called data_lake_ingestor.py, which fetches data from the GitHub Archive for a specific hour. It uses Python's requests library to download the compressed JSON file in memory. The data is then uploaded directly to the specified S3 bucket, with the S3 key based on the date and hour.

To execute the ingestion pipeline all we need to do is passing a timestamp to the ingestion method ingest_hourly_gharchive() , which will be used to determine the interval for the JSON file to collect and load from source.

Managing Configurations and Secrets

Configuration settings, such as AWS credentials and bucket names, are stored in a configuration file (config.ini), to keep the code free from any sensitive and static data.

[aws]
s3_access_key_id = your_access_key_here
s3_secret_access_key = your_secret_key_here
s3_region_name = your_region_name_here
s3_endpoint_url = your_custom_endpoint_url_here

[datalake]
bronze_bucket = you_bronze_zone_bucket
silver_bucket = you_silver_zone_bucket
gold_bucket = you_gold_zone_bucket

In the code, we utilise Python's configparser library to load these configurations into the class, as demonstrated in the private method below.

Raw Data Serialisation - 🥈 Silver Zone

After ingesting the raw GitHub Archive data into our data lake’s Bronze layer, the next critical step in the pipeline is to clean and serialise this data, preparing it for the Silver layer. This is where DuckDB plays a key role, performing the necessary transformations within the data lake.

The transformation logic is encapsulated in the DataLakeTransformer class, located in data_lake_transformer.py. This class provides two primary methods: serialise_raw_data() for data cleaning and serialisation, and aggregate_silver_data() for aggregating the data.

Let’s take a closer look at the serialisation logic:

This method performs several key steps:

Source and Sink Configuration: It determines the source (Bronze) and sink (Silver) bucket names based on the parameters specified in the configuration file (config.ini).

Data Loading: The logic for importing the raw JSON data into an in-memory table using DuckDB’s Relational API is defined in the following method:

The method returns a duckdb.DuckDBPyRelation object, which acts as a relational reference to the in-memory table. This ensures that subsequent steps operate on the in-memory data, avoiding repeated reads from the source file.

A key detail here is the ignore_errors=true parameter. DuckDB infers the schema from the first few records, and in the case of large datasets with inconsistent schemas (e.g., extra nested attributes in some records), errors may occur.

By setting ignore_errors=true, DuckDB skips over records that don’t match the inferred schema, which is efficient for our use case, where we don’t need deep, optional attributes found in some records. Alternatively, we could scan more records or provide an explicit schema, but that would introduce significant overhead for the large files we are processing.

Data Modeling

Before we serialise the raw data into Parquet format we need to perform a data modeling exercise and only select the attributes we are interested in. For this we can use DuckDB SQL to implement the data modeling logic as shown in the following method.

The result of the SQL query encapsulated in the method below, is also stored in an in-memory table, ensuring that multiple future calls do not re-execute the SQL logic.

DuckDB offers powerful features for working with nested JSON files, such as those found in the GitHub Archive dataset. One of the key advantages is the ability to use dot notation to access nested attributes directly, as well as functions like unnest() to fully unpack nested structures in your queries.

For example, if we wanted to flatten and extract all the attributes within the actor object, we could do so with a simple query like:

query=f"SELECT UNNEST(actor),.... FROM '{raw_dataset}'"

This approach makes it easy to work with complex, deeply nested data while maintaining simplicity in your queries.

Data Export: After the data has been cleaned, the result is written to the Silver layer in Parquet format using DuckDB’s engine:

gharchive_clean_result.write_parquet(sink_path)

Serialisation Performance

When running the serialisation process close to the data, and using DuckDB to handle the transformation, the entire process completes in under a minute.

This efficiency makes DuckDB an excellent choice for lightweight, in-place data transformations, especially when working with local or cloud-based object storage systems like S3.

2024-10-01 15:41:04,365 - INFO - DuckDB - collect source data files: s3://datalake-bronze/gharchive/events/2024-10-01/15/*
100% ▕████████████████████████████████████████████████████████████▏
2024-10-01 15:41:29,892 - INFO - DuckDB - clean data
2024-10-01 15:41:30,129 - INFO - DuckDB - serialise and export cleaned data to s3://datalake-silver/gharchive/events/2024-10-01/15/clean_20241001_15.parquet
100% ▕████████████████████████████████████████████████████████████▏

Key Takeaways

The use of DuckDB for this transformation process is a key design choice:

• In-Memory Processing: DuckDB allows for efficient in-memory processing of the data, which is particularly useful for the typically large GitHub Archive datasets.

• SQL Interface: The use of SQL for data modeling provides a familiar and powerful interface for data transformations.

• Parquet Writing: DuckDB has very efficient Parquet reader and writer for fast and efficient serialisation of data from primitive data types such as JSON and CSV, while eliminating the need for intermediate steps or additional libraries.

Data Aggregation - 🥇 Gold Zone

After modeling and serialising our GitHub Archive raw data into the Silver zone, the next step in our data pipeline is to aggregate this data and publish it to the Gold zone on daily basis.

Here’s an overview of the method responsible for performing the daily aggregation:

This method encompasses several key steps:

Source and Sink Configuration: It identifies the source (Silver) and sink (Gold) bucket names based on the configuration.

Data Loading and Aggregation: The aggregation logic is defined by using SQL applied to a DuckDB virtual table that has been defined over Parquet files present in the Silver zone. This aggregation focuses on counting GitHub events by type (e.g., stars, pull requests), repository, and date, providing an aggregated view of GitHub activity within a daily time window.

The GROUP BY ALL feature in DuckDB simplifies group by statements by removing the need to explicitly specify the columns.

As with the previous transformation step, we persist the result of this aggregation to an in-memory DuckDB table, returning it as a DuckDBPyRelation object to ensure that future calls do not re-execute the SQL logic.

Data Export: The aggregated data is subsequently written to the Gold zone in Parquet format:

gharchive_agg_result.write_parquet(sink_path)

Aggregation Performance

The transformation pipeline on a cloud VM takes less than a minute to aggregate 24 Parquet files containing nearly 6 million records, and serialise the result into a Parquet file published in the Gold zone.

This efficiency underscores the capability of DuckDB to handle small to medium-scale data transformations quickly and effectively.

2024-10-01 00:31:42,787 - INFO - DuckDB - aggregate silver data in s3://datalake-silver/gharchive/events/2024-10-01/*/*.parquet
100% ▕████████████████████████████████████████████████████████████▏
2024-10-01 00:31:53,020 - INFO - DuckDB - export aggregated data to s3://datalake-gold/gharchive/events/2024-10-01/agg_20241001.parquet
100% ▕████████████████████████████████████████████████████████████▏

Key Takeaways

The use of DuckDB for this data aggregation process offers several advantages:

• Efficient Processing: DuckDB's column-oriented storage and processing is well-suited for analytical queries and aggregations.

• SQL Interface: The use of SQL for aggregation in Python provides a familiar and powerful interface for complex data transformations.

• Efficient Parquet Integration: DuckDB's native support for Parquet files allows for efficient reading and writing of data in this format.

Orchestration & Scheduling

In a production environment, you would typically use a workflow orchestrator like Apache Airflow, Dagster or Prefect to manage the execution of the three pipelines discussed in this article.

However, since the goal here was to demonstrate how DuckDB can be used for data transformation, I’ve intentionally omitted any external orchestration or scheduling components. Instead, in the GitHub project, I’ve provided individual scripts for each pipeline along with instructions in the README on how to schedule them easily using cron.

That said, you can easily adapt the code to your preferred workflow orchestrator. For example, in Airflow, you could use a PythonOperator to call the functions for each pipeline step by importing the relevant class.

This code-first approach to pipeline development ensures that your business logic remains decoupled from workflow logic, making the pipelines flexible and easy to port to any Python-based orchestration tool.

Interactive Data Analytics

Once the data is prepared for analytics, DuckDB's efficient querying capabilities allow us to easily extract meaningful insights from the aggregated data, making it an excellent tool for interactive data analysis.

Following code snipped from a Jupyter notebook on my laptop demonstrates how to analyse the most-starred repositories on a specific day using the Parquet files stored in the Gold zone.

The code showcases DuckDB's Python API, highlighting its Pythonic data analysis capabilities, which are comparable to other DataFrame APIs like PySpark and Pandas.

Conclusion

In this article, we explored how DuckDB can be leveraged for efficient batch data transformation and serialisation in a data lake architecture. We walked through the steps of ingesting raw GitHub Archive data, transforming it into structured formats, and aggregating the data for analysis using Medallion architecture (Bronze → Silver → Gold).

DuckDB’s in-memory processing, SQL-based transformation capabilities, and seamless Parquet integration make it an excellent choice for handling such datasets with high performance.

For those looking to replicate this approach, the full code examples are available on GitHub, allowing you to explore the potential of DuckDB in your own data transformation pipelines.

Follow Me on LinkedIn

✨ Featured - The New Open Catalog War

In July this year, Snowflake open-sourced its Polaris Catalog under the Apache 2.0 license, with plans to submit it to the Apache Incubator program. Polaris is a catalog service designed for Apache Iceberg but can extend to other major open formats as well.

The question is: why does Apache Iceberg, which already has its own metadata layer, need a separate catalog service?

While Delta Lake, Apache Hudi, and Apache Iceberg open table formats provide their own metadata, each query engine (such as Spark, Flink, Presto or Trino) must perform separate integrations for tasks like schema discovery and data operations.

An open unified catalog service like Polaris simplifies this by streamlining multi-engine interoperability. It also offers enhanced features like improved search, data discovery, tagging, and governance, including access control through a unified, open, and vendor-agnostic interface compatible with various storage engines.

Polaris has the potential to become the standard catalog service for data lakehouse platforms, much like Hive Metastore was for Hadoop-based systems. Currently, catalog options include proprietary tools like AWS Glue Catalog and Databricks’ Unity Catalog, which was also open-sourced in June 2024.

Snowflake’s decision to open-source Polaris may have been influenced by Databricks' move to open source Unity Catalog. While some may see this as a new “catalog war” driven by marketing strategies, as noted by Chris, there’s still hope that these moves will lead to production-ready, open-source catalog services that can finally provide an alternative solution to Hive Metastore after all these years.

📡 Open Source News

👉 Seamless Integration of dbt and Airflow

Building and scheduling data pipelines using dbt models and workflow orchestration tools like Airflow has become a standard practice in data engineering for running transformation workflows. In response to the growing demand for seamless integration between the two systems, Astronomer developed a Python package called Cosmos. This package simplifies running dbt models within Airflow DAGs through a new Airflow task type, DbtDag. The latest version, 1.5.1, was released on July 17, so if you're using dbt and Airflow, it's definitely worth checking out. —> Read More

👉 What's New in Apache Kafka 3.8.0

The release of Apache Kafka 3.8.0 was recently announced, bringing several new features and improvements. This post on the official Apache Kafka site provides an overview of key updates, including support for compression levels, a new consumer rebalance protocol, and re-bootstrapping capabilities. —> Read More

👉 Apache Flink 1.20: New Features and the Road to Flink 2.0

Apache Flink 1.20 was also recently released, and the Confluent blog highlights the major improvements and features in this update. Notable enhancements include improvements to the bucketing feature for Flink SQL tables, allowing users to specify the number of buckets in the DISTRIBUTED BY clause, and the introduction of Flink SQL Materialised tables, which are automatically refreshed in the background as data streams in. Additionally, there are various operational improvements. According to reports, this may be the last minor release before Flink 2.0. —> Read More

🛠 Practical Data Engineering

👉 Mastering Airflow Trigger Rules

Astronomer, a managed Airflow service, has provided an overview of Airflow trigger rules with a visual guide to help new Airflow developers better understand and apply the right trigger rules in their DAGs. For new engineers, grasping all the trigger rules can be challenging, but it's a crucial aspect for effectively managing dependencies between upstream and downstream tasks. —> Read More

👉 A Radical Simplicity Approach to Data Engineering

The tds blog recently shared a great post about the trade-off between simplicity and functionality in software projects, including data engineering. The author advocates for a philosophy of "Radical Simplicity," where simple, straightforward solutions are prioritised over complex ones. This resonates with me, as I believe complexity should only be introduced when absolutely necessary. —> Read More

👉 Hands-On Guide: Installing and Integrating Polaris OSS

If you're interested in installing and testing the latest Apache Polaris open-source release, Dremio has published a hands-on tutorial that guides you through the installation process and integration with Apache Spark and Apache Iceberg. —> Read More

⚙️ Technical Deep Dive

👉 Parquet vs ORC: Choosing the Right Format for Data Lakehouse

In a recent Apache Hudi blog post, the author compares Parquet and ORC, two of the most popular serialisation frameworks for data lakes and open table formats. The post argues that Parquet delivers better performance for read-heavy, complex analytical use cases where query performance is crucial, while ORC offers a more balanced approach for both read and write performance with superior compression, making it a better fit for general-purpose data storage in Hudi. —> Read More

👉 Overview of Amazon MSK Tiered Storage

Amazon recently published a post, explaining how the new Kafka tiered storage in Amazon MSK (Managed Kafka Service) enhances scalability and resiliency. With the new decoupled storage and compute architecture, the system benefits from faster broker recovery, improved load balancing, and virtually unlimited scalability. —> Read More

💬 Community Discussions

There have been several widely discussed threads including link and link on Reddit last month about data engineering roles and the challenges of applying for Data Engineering jobs. Based on comments from both candidates and hiring managers (including CTOs), the situation seems two-fold.

On one hand, the market is flooded with unqualified candidates—often with minimal skills from low-quality bootcamps and online courses—making it hard for companies to find qualified engineers. On the other hand, there are vague and unclear job descriptions, leaving data engineers unsure of what is expected of them once hired. This has created frustration on both sides.

🔎 Case Studies

👉 Evolution of Apache Flink Architecture at Airbnb

Airbnb published a post detailing the evolution of their Apache Flink architecture. Initially, they deployed Flink jobs on Hadoop YARN with Airflow as the scheduler in 2018. Today, they’ve moved to deploying Flink jobs on Kubernetes, eliminating the need for a job scheduler. —> Read More

👉 Pinterest's Adoption of StarRocks for Real-Time Analytics

Pinterest shared their recent adoption of StarRocks, a real-time OLAP engine, for their real-time analytics platform. They chose StarRocks for its features like support for standard SQL, joins, sub-queries, and materialized views—capabilities not readily available in other real-time OLAP engines like Druid. Back in 2021, Pinterest published details about managing a large Druid fleet with 2,000 nodes in a multi-cluster setup. —> Read More

👉 The Rise of New Real-time OLAP Engines

On that note, while Apache Druid, Pinot, and ClickHouse have dominated the open-source real-time OLAP space in recent years, we’re now seeing increased adoption of newer engines like Apache Doris and StarRocks, the latter being a fork of Doris. For a detailed comparison between StarRocks and Doris, check out this blog post from StarRocks Engineering.

👉 Uber’s Hadoop-to-Cloud Migration

Uber, which operates one of the largest on-premise Hadoop clusters, has recently begun migrating to the cloud, starting with a key architectural shift—replacing the HDFS file system with Google Cloud Storage, while still running the rest of their stack on IaaS. One of the challenges in migrating from Hadoop to the cloud is transitioning Hadoop’s security features, such as delegation tokens and Kerberos authentication, to Google Cloud’s token-based security. Uber discusses how they tackled these security migration challenges in this article. —> Read More

📣 Vendors News & Announcements

👉 Fivetran Integration with Snowflake’s Polaris Catalog

Just days after Snowflake open-sourced the Polaris catalog service on GitHub, Fivetran, a leading SaaS provider for data integration, announced its upcoming integration with the newly open-sourced Polaris data catalog. This integration aims to develop a managed catalog solution for Fivetran’s Managed Data Lake Service. —> Read More

👉 Databricks LakeFlow Connect for Automated Data Ingestion

Databricks announced the public preview of LakeFlow Connect, an automated incremental data ingestion service for sources like SQL Server and Salesforce. Built on Delta Live Tables, LakeFlow Connect enables incremental data ingestion using CDC (Change Data Capture). This marks another step by major vendors toward automating data engineering tasks. —> Read More

👉 Databricks Lakehouse Federation Across AWS, Azure, and GCP

Databricks also announced the general availability of Lakehouse Federation in Unity Catalog across AWS, Azure, and GCP last month. This mirrors the strategy of other top cloud vendors to offer a unified analytical platform with centralised data discovery and governance, providing a unified view of enterprise data across multiple storage engines and cloud platforms. —> Read More

👉 ClickHouse Acquisition of PeerDB for Real-Time Postgres Ingestion

ClickHouse, Inc. announced the acquisition of PeerDB, a provider of Change Data Capture (CDC) for Postgres databases. This move aims to integrate and streamline real-time data ingestion from transactional databases like Postgres into the ClickHouse OLAP engine. —> Read More

🎥 Conferences & Events

👉 Kafka Current 2024 Key Notes

The two-day Kafka Current 2024 event, organised by Confluent, took place last week in Austin. Keynotes from both Day 1 and Day 2 have already been published on YouTube: Keynote Day 1 | Keynote Day 2

👉 Open Source Data Summit Virtual Conference

The Open Source Data Summit Virtual Conference will be held on October 2nd. If you're interested, you can register for free at opensourcedatasummit.com. The event will feature numerous discussions on data lakehouses and the role of open table formats in modern data architectures.

After years of working in the data space, you witness the rise and fall of numerous tools and products. Over time, you become more skeptical of the latest shiny tools, often dismissing them as marketing-driven hype, until you reach a point where they can no longer be ignored.

DuckDB was one such tool for me. Initially, I dismissed it, but eventually, the buzz around it became too loud to ignore. That's when curiosity got the better of me, and I decided to dive in.

But like some new technologies, I initially struggled to grasp its core functionality. The official website describes it as a "fast in-process analytical database" but this wasn't very enlightening.

Then I came across the phrase “SQLite for OLAP” which helped me understand it better, though I was still unsure about its place in my mental data stack.

It wasn’t until I explored discussions in data communities like Reddit that I realised DuckDB, like many open-source projects, had evolved beyond its creators' original vision. That’s when I knew it was time to test it out for myself and decide if the hype was justified.

WHAT DUCKDB IS

In my findings, DuckDB is a hybrid engine with a range of diverse functionalities. Let's explore these features in more detail.

An Embeddable & Portable Database

DuckDB is an "embeddable" database system. This is what the creators of DuckDB described it as, in a SIGMOD conference paper back in 2019.

Like SQLite, it allows you to store your data in a single .duckdb database file, making it easily portable within other projects.

Besides being embeddable, DuckDB operates much like traditional DBMS systems, but without requiring a long-running server process like MySQL or PostgreSQL. There’s no need for starting a database server and establish a socket connection to it over IP/hostname and port.

All you need to do is point to where the single DuckDB database file is stored, or start an in-memory session without even requiring to use a physical database.

A Columnar OLAP Database

DuckDB is a columnar database, making it highly efficient for running analytical queries.

It supports two main storage formats: its native .duckdb format or open-standard file formats like Parquet, which DuckDB reads and writes with impressive efficiency, including support for predicate pushdown.

Internally, DuckDB uses a row-columnar structure. Data is sliced into row-groups containing 120,000 records, and within each group, columns are stored separately and compressed—similar to popular binary formats like Parquet and ORC.

This architecture allows you to efficiently store and query analytical datasets on a local machine or a single server without the overhead of a full-fledged OLAP database engine.

Interoperable SQL-Powered DataFrame

The DuckDB’s Python library is essentially a DataFrame on steroids. It integrates seamlessly with popular DataFrame libraries like Pandas and Polars, allowing efficient in-memory operations.

What sets DuckDB apart is its ability to run SQL queries directly on Python DataFrames. You can query Pandas, Polars and Apache Arrow DataFrame objects as though they were SQL tables.

For some frameworks such as Apache Arrow, DuckDB uses zero-copy mode for fast conversion. Using zero-copy mode no serialisation is required for translating the in-memory objects between different representations.

You can join data from different objects, such as a Polars DataFrame, and a Pandas DataFrame, in a single SQL query. The results can be stored back into a DuckDB database or exported to file formats like Parquet on external storage.

A Federated Query Engine

DuckDB offers a simple, efficient way to query external data systems through its extensions. Similar to distributed query engines such as Athena, Presto or Trino, it allows seamless joins across various external data sources.

You can directly query DBMS systems like MySQL and Postgres, open data files like JSON, CSV, and Parquet files stored in cloud storage systems like Amazon S3, and modern open table formats like Apache Iceberg and Delta Lake.

Though DuckDB doesn't have the concept of external tables found in systems like Hive or Redshift, you can still create persistent Views over the external tables or data files, like a read-only external table.

Querying a remote file in cloud storage can be done without downloading the entire file. DuckDB efficiently samples and inspects data.

A Single-Node Compute Engine

DuckDB can also act as a single-node compute engine, performing ephemeral batch transformations. It’s like having a stand-alone Spark at your disposal for smaller-scale workloads.

This is particularly useful in scenarios like data lake architectures, where DuckDB can efficiently serialise raw data (e.g., JSON or CSV) into optimised formats like Parquet, and then transform or aggregate that data.

I tested this functionality implementing a simple data lake following medallion architecture, using GitHub events data dumps (known as GH Archive) as the data source.

I setup an hourly data ingestion pipeline to collect and load hourly archives from gharchive.org server into my data lake’s Raw zone on S3.

I then created an hourly transformation pipeline using DuckDB to easily unpack, clean and serialise the hourly JSON dumps into Parquet formats published to my Silver (clean) zone.

Finally, I used DuckDB again to run a daily aggregation job to aggregate the Github events grouped by event type, repository, and event date, and export the result to the Gold (analytics) zone.

The following code sample demonstrates the aggregation logic. On my laptop, DuckDB reads and processes 24 compressed Parquet files from S3, containing a total of approximately 5 million records, and exports the results back to cloud in about 5 minutes.

The result was a simple and efficient data pipeline, and anlaytical-ready datasets in Parquet format that could be queried from my local machine using DuckDB’s SQL or Python API.

The following shows the result of the query to get the top GitHub repositories receiving the most stars on 2024-08-27, using DuckDB's Python API on my laptop:

DuckDB might not be able handle terabyte-scale data like distributed engines, but many use cases don’t require such scale. Besides, modern servers are powerful enough to handle a lot of the heavy lifting without needing distributed processing frameworks.

Given the above capabilities, what would you call a system which is:

An embeddable and portable DBMS, columnar OLAP database, a SQL-based interoperable DataFrame, federated query engine, and a single-node compute engine?

Let's call it DuckDB!

WHAT DUCKDB CAN BE

Now that we've explored what DuckDB is, let's dive into its future potentials, examining how it can evolve within the broader data engineering and data science ecosystem.

The Missing Piece in the Data Scientist's Toolbox

DuckDB fills a crucial gap in the data scientist’s toolbox: a powerful in-process database engine that integrates seamlessly into data science workflows without the need for complicated setups, installations, or data transfers between Python and external databases.

According to Mark Raasveld, one of DuckDB's creators, this was one of the primary design goals for building it.

Data scientists often find the process of setting up and using external databases cumbersome. They prefer working with plain-text or binary data files (like CSV or Parquet) for the ease of immediate access and manipulation.

However, DuckDB changes this dynamic by enabling data scientists to work with a relational SQL database directly within their Python environment, eliminating the need for external database installations and the associated data transfer overhead.

Bring Your Own Compute

As personal computers and laptops become more powerful, DuckDB enables a "bring-your-own-compute" model. It allows users to leverage their local machines to run analytics on shared data, whether that data is stored locally or in the cloud.

Users can easily attach to data files or external database system hosted in the cloud and perform their analyses using DuckDB's local processing power. This is especially beneficial for collaborative projects, such as those involving research teams.

An alternative hybrid model is also possible, where part of the data processing happens in the cloud while another part is handled locally. This approach, pioneered by MotherDuck, combines cloud scalability with the efficiency of local compute power.

A Standard Format for Publishing Relational and Analytical Datasets

The DuckDB format (.duckdb database file) has the potential to become the standard for sharing relational data on the internet.

Instead of exporting multiple data objects as separate CSV files, dataset publishers can export a complete dataset as a single duckdb file, containing all the data in an optimised, ready-to-analyse columnar format.

This approach streamlines data sharing and ensures that the data is structured for efficient querying and analysis right out of the box.

Expand DataFrames' Footprint in Data Engineering

The DuckDB Python package bridges SQL and Python, allowing users to query Python in-memory objects like Pandas DataFrames as if they were database tables.

This capability makes DataFrames accessible to a wider audience, including those more comfortable with SQL than Python. It also enhances the use of DataFrames for building data transformation pipelines using SQL.

By leveraging DuckDB’s SQL interface, users could even apply transformational models like dbt on top of their DataFrames, further extending the role of DataFrames in data engineering workflows.

Turn OLTP Databases into HTAP

The recent introduction of pg_duckdb and also pg_analytics is another exciting development.

With pg_duckdb Postgres extension, you can run analytical queries directly within Postgres database without needing to offload data to separate analytical systems.

With pg_analytics extension it’s possible to directly run queries over datasets stored on object stores like S3, and open table formats like Iceberg or Delta Lake directly from Postgres, using the embedded DuckDB as the underlying query engine.

Cloud ProstgreSQL providers such as Crunchy Data are already integrating and taking advantage of DuckDB to extend their service offering for OLAP and analytical use cases.

In-Browser SQL Analytics

In-browser analytics are becoming a reality, thanks to DuckDB’s WebAssembly implementation (DuckDB-WASM). This brings SQL-powered analysis directly into the browser, enabling data analysts to interact with datasets using SQL without installing any software.

It also offers a cost-effective solution for organisations that expose public datasets by allowing the analysis to be done client-side, reducing server load.

For instance, Hugging Face recently introduced a Datasets Explorer Chrome extension, allowing users to explore their datasets using SQL directly in the browser—powered by DuckDB’s engine under the hood. This innovation makes analysing public datasets easier and more accessible than ever.

As we've explored, DuckDB is far more than just another database engine—it unlocks new possibilities for data storage, analysis, and computation.

If you've discovered additional capabilities or unique use cases for DuckDB, I'd love to hear your thoughts in the comments.

Follow Me on LinkedIn

In previous generation Hive-style directory-oriented table format, which I have covered their evolution in a comprehensive essay, data files and partitions were tightly coupled to the underlying physical storage system API.

The query engines relied on the file system API for listing data files and partitions each time a query was executed. Additionally, in systems like Hive an external Metastore is employed for keeping track of metadata such as table partitions.

The modern data lakehouse engines represents an alternative mechanism for implementing table formats in cloud storage systems. Modern table formats maintain the entire table state within log-oriented metadata files.

In this article we will explore the overall design and architecture of the metadata layer on these modern table formats to better understand how they are structured and managed.

Metadata Structure

The design and organisation of the metadata layer can have a significant impact on the metadata access strategy during query planning, which in turn can affect overall query performance. A straightforward approach involves storing all metadata for each dataset in a dedicated special directory which you could think of it as a sub-table inside the main table.

This sub-table, containing the metadata records in a semi-structured log files, is stored alongside the actual data files within the data lake, typically as a subdirectory directly under the root dataset path.

Lets say we have a table called mytable, directly under the main path we can store our special sub-table called ../metadata_logs as a subdirectory containing the metadata logs.

/mytable/
   /metadata_logs/
	000001.log
	000002.log
	000003.log
	000004.log

Together with the data files and partitions the overall structure of a simple table would look like the following:

/mytable
     /ts=2024-08-01/20240801-01.parquet
	          20240801-02.parquet
	          20240801-03.parquet
     /ts=2024-08-02/20240802-01.parquet
		 20240802-02.parquet
     /metadata_logs/000001.log
		 000002.log
		 000003.log
		 000004.log

At the time of the query engine would scan the available metadata logs stored under the ../metadata_logs subdirectory during query planning phase to build a list of actual data partitions and files to be scanned.

How is this more efficient?

This design fundamentally eliminates the dependency on the underlying storage metadata API for performing directory (i.e., partition) and data file listings during the query planning phase—a process that often incurs higher latency and can become a significant performance bottleneck in large-scale data lakes.

Instead, the approach leverages the storage's fast sequential I/O capabilities to scan and read a few metadata log files end-to-end. This method offers significantly better performance compared to gathering file lists and statistics by issuing many LIST API calls.

These API calls are often subject to throttling and limits, typically returning only 1,000 objects per call on cloud platforms. This can be particularly problematic when dealing with potentially hundreds of thousands of files and numerous partitions in large-scale data lakes.

In a typical Hive or Spark workload that needs to perform a query on a table containing couple of years of data partitioned by date, the engine would have to make thousands of API calls for gathering details of all available partitions and their data files in order to perform split planning for assigning work to each worker.

However, if all the details of all data files and partitions are stored in few metadata log files, the engine can quickly scan the logs using fast sequential I/O to identify all the required objects, which is substantially faster.

Based on industry benchmarks, the new approach significantly reduces the directory and file listing latency compared to direct listing using the underlying storage API such as HDFS and S3.

Hudi's experiments shows 2-10x improvement in listing latency compared to S3 over large datasets.

Hierarchical Metadata Organisation

Another way to structure the metadata layer is by using a hierarchical approach. In this method, the lower levels of the hierarchy consist of files that store metadata about specific sets of data files. As you move up the hierarchy, index files aggregate metadata from the layers below, effectively serving as a table index.

In Apache Iceberg’s layered design, at the lowest level, each metadata file (refered to as manifest file in Iceberg terminology) would track a subset of the available data files for a table.

In the second layer, a Manifest list file contains a high-level index of the collection of the manifest files in the lower layer. It stores essential details such as the current snapshot ID, the locations of the manifest files, and partition boundaries.

Finally, a master Metadata File sits at the top of the hierarchy as shown below, storing a high-level snapshot view of the table's metadata. This file is referenced by the table's metadata pointer attribute and is regenerated whenever there is a change in the table's metadata.

Importantly, this hierarchical structure of metadata does not need to correspond to a physical hierarchy in terms of file organisation. For example, in Iceberg, all metadata files are kept under a single metadata directory, regardless of their hierarchical relationships

Why would Iceberg structure the metadata in this way?

The purpose of designing a hierarchical indexing structure in Apache Iceberg is to optimise metadata lookup performance by minimising the number of metadata files that need to be scanned during query execution.

By organising metadata in a more structured and layered manner, the system can quickly locate relevant data, leading to faster query response times. However, the trade-off is a more complex metadata structure compared to the simpler, flat design.

Metadata Storage Models

The storage model is concerned with how the metadata records are managed in the metadata log files.

An important design factor to consider is that due to the immutable nature of the underlying storage layer, metadata update events—such as the addition or removal of data files—cannot simply be appended to the existing metadata files. As a result, for new metadata update operations a new delta file must be generated.

To ensure storage and I/O efficiency, the frameworks typically perform a periodic background compaction operation. This process merges smaller delta logs into a snapshot base log. At any given point in time, the table's metadata is a combination of the last snapshot log and any new delta logs that have not yet been compacted.

There are two main storage models currently implemented by the open table formats for storing and managing table's metadata:

Log-structured Metadata Model

This is the simplest implementation technique where all metadata changes are treated as immutable, ordered events stored sequentially in transactional event logs.

This approach essentially applies the event sourcing pattern to capture all state changes at the file level, recording them in transactional logs that are stored alongside the actual data files similar to WAL implementation of the database engines.

Files and partitions serve as the unit of record for which the metadata layer tracks all state changes, capturing these changes within the event log. To rebuilt the current state of the table, all records in available logs are scanned sequentially from top-to-bottom.

Instead of using an unstructured format, semi-structured formats like JSON can be employed to provide better schema support and flexibility, while being both human-readable and easily parsed by machines. This is the approach used by Delta Lake, where transactional metadata logs are stored in JSON format.

Table-oriented Metadata Model

In an object or table-oriented metadata design, the framework treats table metadata—primarily dataset partitions and file listings—as a special "table" stored in a more structured file container, similar to how it handles the base table containing the actual data files.

The key difference from the framework's perspective is that this special table is used to manage metadata events rather than actual data records.

What are the trade-offs between the two models?

Compared to a log-structured implementation, a table-oriented metadata design is a relatively more closed and abstract approach to managing the metadata layer in open table formats.

In a log-structured approach, transactional logs are the first-class citizens of the metadata layer, directly accessible by the framework as well as other engines and users. In contrast, the table-oriented design uses a logical table as the access point for all metadata operations, abstracting the underlying metadata log files.

A Combination of the two models are used by the existing frameworks. Apache Hudi heavily relies on optimised table-oriented metadata management using HFile structured file format. The metadata table is internal to the framework and not exposed so much to users or clients.

On the other hand, Delta Lake and Apache Iceberg follow more open log-oriented design offering the additional benefit of supporting metadata streaming and event-based ingestion primitives out of the box.

Metadata Index Files

In this section, we will explore the format of the different metadata files used in the metadata layer, and the how the table events are captured and managed.

Data Files Index

A data file index is a type of manifest file used to maintain the list of active data files associated with a table.

As mentioned earlier, the goal is to eliminate the need for performing recursive storage metadata API calls to gather the list of data files. A file index can also help reduce cost on cloud object stores by significantly lowering the rate of API calls required during reading and writing operations.

Implementing a file index for a non-partitioned table is relatively straightforward. A sequential log file at the base table level can be used to insert new filenames—either with full or relative paths— to a metadata log. During the query planning phase, the generated log files can be scanned from start to finish to identify all active files belonging to a table.

For capturing storage-level or filesystem state changes we need to consider two main filesystem object types, that is files and directories (i.e partitions) with following possible events:

In a most simple form, the files index can be implemented with following fields in a WAL (Write Ahead Log) type file:

timestamp|object|event type|value
20231015132000|partition|add|/year=2024/month=08/day=15
20231015132010|file|add|/year=2024/month=08/day=15/00001.parquet
20231015132011|file|add|/year=2024/month=08/day=15/00002.parquet
20231015132011|file|add|/year=2024/month=08/day=15/00003.parquet

Implementation of Data Files Index

Delta Lake manages file listing information at the base level of the table using a combination of a base Parquet file and additional JSON transaction log files. All file-level updates are committed to the JSON files at the time of writing.

Apache Hudi maintains all file listing metadata in a single base HFile, stored under the .hoodie/metadata/files path. The HFile is partitioned into four sections: files index, column stats, bloom filters, and record level index. The Files index contains two types of records: one for adding new partitions and another for adding new files.

Following shows an example of adding a new partition key 2024-08-01 with a new file file10.parquet:

-- Insert a new file record (Type 2): 

{"key": "2024-08-01", "type": 2, "filenameToSizeMap": {"file10.parquet": 12345, isDeleted: false} }

-- Insert a new partition record (Type 1):

{"key": "_all_partitions_", "type": 1, "filenameToSizeMap": "2024-08-01": 0, isDeleted: false} }

Why Hudi selected HFile as the metadata file format?

The motivation behind selecting HFile file format is that even for very large datasets containing thousands of partitions and millions of files, the expected compressed and encoded HFile would be within a manageable size, typically hundreds of megabytes and less than a gigabyte.

Additionally, having fewer metadata files to scan can significantly improve performance during read operations. Consequently, HFile has been selected for managing both base and log files within the metadata layer by Hudi.

Hudi claims that based on experiments, HFile performs better than Avro and Parquet (Used by Delta and Iceberg) for point lookups (10x to 100x improvements) over large number of metadata entries.

Snapshot File Index

Periodically, a log checkpoint operation is performed on a specific set of delta log files to summarise the log up to that point into a snapshot file. This process eliminates duplicate entries and retains only the latest list of active partitions and files. As a result, the checkpoint file represents the state of the table at a specific point in time.

In Delta Lake, the snapshot file is maintained in Parquet format. To compute the current table state in terms of active files and partitions during a query, the latest snapshot Parquet file, along with any subsequent JSON transactional logs, must be consulted to obtain the current list of active files in the table.

Example of an ‘add’ action log record in Delta:

{
  "add": {
    "path": "date=2024-08-10/part-000...c000.gz.parquet",
    "partitionValues": {"date": "2014-08-10"},
    "size": 841454,
    "modificationTime": 1512909768000,
    "dataChange": true,
    "baseRowId": 4071,
    "defaultRowCommitVersion": 41,
    "stats": "{\"numRecords\":1,\"minValues\":{\"val..."
  }
}

Managing Deletes

When files or partitions are deleted, the file index needs to track these changes and remove the corresponding listings. Due to immutable nature of the log files, the file index cannot be simply updated to remove the entries.

A common approach to handling deletions is the use of a delete tombstone method, where a new record with a delete marker is inserted into the metadata log. This marker is then used during query time to identify and filter out the deleted files.

Additionally, a background compaction operation can be employed to clean up the file index by removing duplicate keys and older versions of records, thereby maintaining an up-to-date and efficient index

Hudi uses a special field in each record inserted into the HFile index called ‘isDeleted’ as the delete tombstone hint:

{"key": "2024-08-12", "type": 2, "filenameToSizeMap": {"file20.parquet": 65432, isDeleted: true}}

Delta Lake follows similar delete tombstone technique, however unlike Hudi which uses an attribute in the inserted log as a delete hint, Delta Lake uses a distinct metadata action for the transaction log called ‘remove’ when committing a new log to the transaction log.

During log compaction process previous duplicate ‘add’ actions are eliminated but their ‘remove’ actions are maintained until the retention period has expired.

Example of a ‘remove’ action log record in Delta:

{
  "remove": {
    "path": "part-00001-4.parquet",
    "deletionTimestamp": 1515488765783,
    "baseRowId": 1171,
    "defaultRowCommitVersion": 11,
    "dataChange": true
  }
}

Unlike the previous two implementations, Apache Iceberg maintains a separate manifest file which only contains deleted files logs. At read time, the delete manifest files are scanned first during query planning to identify which files to exclude during query planning phase.

Column-stats Index

An optimisation feature of open table formats is the storage of table statistics, such as column min-max values, in a separate metadata structure to facilitate faster query planning.

The goal is to eliminate the need for the conventional, expensive per-object statistics gathering during the query planning phase. This typically involves scanning the footer sections of all Parquet or ORC data files to prune irrelevant data files. This approach is not scalable when dealing with a large number of partitions and data files, as a large query would require scanning each serialised file's footer separately, resulting in numerous I/O calls.

By storing column statistics in a separate, efficient metadata index file, query engines can leverage faster and more efficient sequential I/O to perform the necessary query planning, significantly improving performance

Implementation of Column-Stats Index

Delta Lake supports storing per-column statistics within the same JSON-based delta log files, using a nested structure within the main data file metadata record. This structure mirrors the schema of the actual data by storing each file's minimum and maximum statistics.

For example, if the schema contains columns like product and price, the ‘stats’ key in the metadata structure of the delta log would resemble the following format:

{
  "add": {
    "path": "date=2014-08-10/part-000...c000.gz.parquet",
    "partitionValues": {"date": "2024-08-10"},
    ...
    "stats":{
         "numRecords": 10,
	...
	"minValues": ["product": "book","price": 15],
	"maxValues": ["product": "book","price": 23]
    }
}

Apache Iceberg's design for storing both table and column-level stats is similar to Delta, by storing each type of statistic such as column min and max values, in a separate field of array type, inside the main data_file metadata record of manifest files.

To make the column stats object more compact, Iceberg stores the column id as the reference instead of using the column name

"value_counts": [
     {
	"key": 1,
	"value": 34535
     },
     {
	"key": 2,
	"value": 54321
     },
     {
	"key": 3,
	"value": 5566712
     },	
]

Since version 0.11, Apache Hudi introduced a multi-modal index, which also stores column-level statistics within the same metadata table, specifically in the HFile metadata log.

These statistics are stored under a separate partition (i.e., key) as a stand-alone partition called column_stats. By keeping the keys sorted, this structure leverages the locality of all records for a particular column, enhancing query performance and efficiency.

Since the statistics are stored under a separate key, each record must include the filename and path fields which can be seen as an extra overhead compared to Delta and Iceberg implementation. The current supported statistics include minValue, maxValue, valueCount, and nullCount.

Unlike Delta and Iceberg, which store statistics differently, Hudi uses an additional field, ‘isDeleted’, as a marker to indicate whether a column stats record is valid or not, due to the stats being stored under a separate partition.

Advantage of combining different index types in a single object

Using a single index file to manage both types of metadata (files and column stats) offers the advantage of reducing both metadata file management complexity and I/O overhead. Additionally, employing a single object with nested structures to store additional metadata allows for the use of a unified schema to manage the table's metadata.

This approach eliminates the need to manage multiple schemas for each metadata type, simplifying overall metadata management. Furthermore, there is no need to separately track the statistical metadata of deleted data files, as the unified structure inherently handles this

Partition Management

As open table formats represent an evolution of data lake formats by introducing a metadata abstraction layer on top, they inherently support conventional hierarchical partitioning schemes on data lakes.

Unlike traditional Hive-style partitioning, where partitions are physically represented by a directory structure, open table formats treat these directories as configurations.

These configurations provide instructions to the processing engine on how to structure the data and maintain the relationship between data files and the projected structure. The partitions themselves are stored and tracked in the metadata layer, either as a separate structure (index) or embedded within the entries of the main file index log.

By managing the mapping of files to partitions in the metadata layer, this design effectively decouples physical partitioning from logical partitioning at the table level.

This decoupling allows for greater flexibility, supporting partition evolution, partition consolidation, and the possibility of utilising different partition schemes within the same dataset.

Moreover, this approach abstracts the partitioning scheme from ETL and data ingestion pipelines. It eliminates the need for these pipelines to be aware of the table's partitioning scheme, allowing for either explicit or dynamic partitioning in SQL transformation statements.

This abstraction also mitigates some of the traditional trade-offs, such as the challenges associated with using event time for partitioning, particularly in handling late-arriving events.

Another notable technique employed by table formats like Apache Iceberg involves automatically generating and using hash-based paths in cloud object stores. This strategy distributes files within a partition across multiple prefixes, helping to avoid throttling and performance issues when scanning a large number of files, especially due to per-prefix call limits.

Implementation of Partition Management

In its simplest form, a new log entry can be added to the file index for each partition added to the base table within a log-oriented structure. This approach is akin to how Delta manages metadata updates.

Specifically, Delta captures any changes to a table's metadata—such as schema modifications and updates to the partitioning scheme—within a ‘metaData’ action type (a struct) as an entry in Delta's transaction logs. The ‘partitionColumn’ attribute within this record captures the current partition columns of the table, ensuring that the metadata reflects the most up-to-date structure.

Here’s an example of the ‘metaData’ action entry in Delta’s log files:

{
  "metaData":{
    "id":"af44c3a7-abc1-4a5a-a3d9-33d65ac711ba",
    "format":{"provider":"parquet","options":{}},
    "schemaString":"...",
    "partitionColumns":["process_date"],
    "configuration":{
      "appendOnly": "true"
    }
  }
}

In an object-oriented metadata structure, all partitions can be encapsulated within their own object (struct) in the main index file. This method is employed by Apache Hudi, where table partitions are tracked by inserting a new log entry into the base HFile index with a specific key, ‘_all_partitions_’.

This key aggregates all partitions within a single object, making it easier to manage and track the partitions at the table level. By using this approach, Apache Hudi efficiently maintains an up-to-date record of all partitions within the dataset.

{
   "key": "_all_partitions_",
   "type": 1,
   "filenameToSizeMap": "2024-08-10",
   "isDeleted": "false"
}

Apache Iceberg also stores partitioning details in the top-level metadata JSON file under its own ‘partition-specs’ key. This key encapsulates the partitioning scheme used by the table, detailing how data is logically divided.

{
    ...
    "partition-specs" : [ {
        "spec-id" : 0,
        "fields" : [ {
            "name" : "process_date",
            "transform" : "process_date",
            "source-id" : 2,
            "field-id" : 1000
        } ],
}

Apache Iceberg goes a step further by storing additional partition statistics such as record count, file count, total size, and delete record count within its metadata. These statistics can significantly enhance query planning and cost-based query optimisation.

Additionally, these statistics are used to facilitate Dynamic Partition Pruning, a technique that reduces the amount of data scanned during queries by eliminating irrelevant partitions early in the query execution process.

What has been presented here forms the foundation of the architecture behind the metadata layer in modern open table formats. However, there are many additional details and advanced features that these systems offer. For those interested in exploring further, the official documentation of the covered projects are a good source of reference.

✨ Featured - Netflix Maestro Workflow Engine

The crowded field of open-source engines has just welcomed a new player: Maestro, recently open-sourced by Netflix!

Netflix asserts that Maestro is a highly scalable and flexible scheduler capable of managing large-scale heterogeneous workflows, including ML training and data pipelines. It supports flexible execution logic, such as Docker images and notebooks, and accommodates various workflow patterns, including cyclic and acyclic (DAGs).

One of its standout features is the foreach pattern, which is particularly useful for repetitive tasks like ML model training and data backfilling—something that would typically require separate job runs on a scheduler like Airflow to backfill daily ingested source data. Maestro also offers multiple domain-specific languages (DSLs) for defining workflows declaratively using YAML files, a feature that would need to be custom-built on top of Airflow.

Since its open-source release in July, the project has already garnered 3,000 stars on Github. Netflix had previously covered the internals and use cases implemented with this engine, and their latest blog post provides a comprehensive overview of Maestro's features and supported workflow patterns.

Additionally, this blog post provides a thorough comparison between Airflow and Maestro, complete with practical examples and code snippets.

💡 Trends & Insight

👉 The State of Modern Data Stack in 2024

In 2024, there has been much discussion about the decline of the Modern Data Stack (MDS). Concerns have been raised about the economics of the Modern Data Stack, and the term itself is being recycled, much like previously hyped concepts such as "Big Data". Some experts believe that many MDS startups are doomed to extinction. As Matt Turck pointed out, the Modern Data Stack was largely a marketing concept and an alliance among several startups across the data value chain. In a recent blog post, Ananth explores the history, decline, and the emerging post-MDS era. —> Read more

👉 Embracing 'Bring Your Own Compute' with DuckDB

An interesting discussion took place between MotherDuck’s co-founder and CEO and Fivetran’s co-founder and CEO about the future of big data and single-node or laptop-sized analytics using DuckDB. With advancements in hardware, we might witness a new shift towards local execution, where computing is fully or partially pushed to the user’s machine, introducing the concept of Bring Your Own Compute for analytics. —> Watch the interview

📡 Open Source News

👉 Apache Kafka 3.8 Release

Apache Kafka 3.8 has been Released. Confluent blog provided a summary of the the new features and improvements in this release. —> Read more

👉 Delta Lake 4.0 New Features

Delta Lake 4.0 Preview was announced in June. A blog post highlights the new important features such as Change Data Feed (CDF), Liquid Clustering and new monitoring features of this release with some practical examples. —> Read More

👉 DAG Factory Project Takeover by Astronomer

Astronomer has announced taking over the open-source project DAG Factory, a Python library for authoring Airflow DAGs declaratively using YAML configuration files. Providing a thin no-code abstraction layer on top of Airflow has become a common practice among tech companies to reduce the engineering effort required to create DAGs, standardise pipeline creation for common use cases like data transformation, and make the process more self-service. —> Read more

🛠 Practical Data Engineering

👉 Consistent Data Modeling and Naming Conventions

Implementing a consistent data modeling framework, such as standardised naming conventions, is crucial to maintaining a healthy data platform and ensuring long-term scalability, even when there is turnover among engineers. Mike discusses some of the key aspects and best practices for data warehouse modeling, including effective naming conventions for tables and schemas. —> Read more

👉 State of CI/CD for Data Pipelines

LakeFS published a comprehensive overview of implementing Continuous Integration/Continuous Delivery (CI/CD) for data pipelines, focusing on the Write-Audit-Publish (WAP) ingestion pattern. The article explores various options and tools available in the market, offering insights into how to effectively integrate CI/CD practices into data workflows. --> Read more

👉 Data Reconciliation Techniques and Best Practices

Datafold has published a three-part series on data reconciliation, a crucial subset of data quality. The series covers use cases, techniques, challenges, and best practices for performing data reconciliation across data sources and targets, with the goal of ensuring data accuracy and completeness. —> Read more

👉 dbt Beyond the Marketing Hype

There have been many blog posts and discussions about the hype dbt has generated over the past few years. The author of this blog post takes a different approach, discussing the challenges of performing data transformation in data warehouses and how dbt can address them. It starts with real problems and then explores how tooling, specifically dbt, provides solutions—rather than starting with the tool (because it's popular and everyone is talking about it) and then searching for problems it can solve. The post also offers a clear definition of what dbt actually does:

dbt works by abstracting common data-warehouse patterns into config-driven automation and providing a suite of tools to simplify SQL transformations, tests, and documentation.

⚙️ Technical Deep Dive

👉 Evolution of Debezium's Internal Engine

While in many streaming CDC data architectures, Debezium plugins are primarily used within the Kafka Connect framework and runtime, it is also possible to run Debezium connectors outside the Kafka ecosystem. This can be done by embedding the Debezium engine in internal applications or by using the standalone Debezium Server, which is now a separate project on GitHub. This Debezium blog discusses the evolution of Debezium's internal engine, starting with the initial EmbeddedEngine implementation, which was mainly built for testing, and the new AsyncEmbeddedEngine, which addresses the shortcomings of the previous implementation. —> Read more

👉 A guide on Concurrency Levels of Apache Airflow

One of the most confusing aspects of the Apache Airflow engine, especially for newcomers, is how concurrency is applied at different levels, such as the Airflow scheduler, DAG level, and task level, and how their combination can impact overall workflow performance. The configuration parameters in the config file of early released versions added to this confusion, with names that often seemed unrelated. In a recent blog post, Google provides a comprehensive overview of the various concurrency levels in Airflow, with a particular focus on its managed Airflow service. —> Read more

👉 Apache Airflow Software Architecture

A helpful guide posted by Apache Airflow's blog with visuals that illustrates the key underlying components of the Apache Airflow software architecture and how they interact within the system. —> Read more

Speaking of Airflow, do you...!?

👉 Bringing GenAI and LLMs to Flink Streaming Pipelines

Confluent blog provided an overview of a new Flink AI feature, which allows streaming data pipelines to invoke AI models, including generative AI (GenAI) large language model (LLM) endpoints (such as OpenAI and Google Vertex AI), directly from Flink SQL statements. This enables tasks like AI model inference, regression, and classification to be seamlessly integrated into real-time data processing workflows. —> Read more

🔎 Case Studies

👉 Pinterest's Migration from HBase to TiDB

Pinterest shared their journey of replacing HBase storage with a modern, scalable open-source database system that meets their requirements for reliability, performance, tunable consistency, and robust CDC support. They ultimately chose TiDB as the solution. We are seeing more stories of companies exploring alternatives to HBase due to its limitations and maintenance overhead. —> Read more

👉 Slack’s Migration to EMR 6

Slack discusses their migration from EMR 5 with Spark 2 to EMR 6 with Hive 3 and Spark 3 on AWS, highlighting the performance and reliability improvements achieved in their data pipelines, which are developed using Apache Spark and scheduled on Airflow. —> Read more

💬 Community Discussions

There was a recent discussion on Reddit on how fast data engineering is progressing. The consensus among most commenters is that while tools, storage systems, and processing frameworks may evolve rapidly, the fundamentals of data engineering remain consistent. These fundamentals include data integration, data modeling, and the processes of extracting, transforming, and loading data (ETL).

For aspiring data engineers, the key takeaway is to invest time and effort in mastering these basics and fundamentals rather than focusing solely on becoming an expert in specific tools. While vendors are striving to automate the data engineering lifecycle as much as possible (as seen with the latest Databricks offerings), a strong understanding of the fundamentals will always be valuable and help you stand out in the field.

📣 Vendors News & Announcements

👉 Snowflake's New Cortex Search Feature

Snowflake announced a new feature called Cortex Search (currently in Public Preview) in July 2024. This search service is designed for unstructured data, such as text, and enables enterprises to deploy Retrieval-Augmented Generation (RAG) applications using Snowflake, allowing them to customise generative AI applications with proprietary data. —> Read more

👉 Databricks Mosaic AI Model Training

Around the same time, Databricks announced the support for Mosaic AI Model Training, which streamlines the fine-tuning of general-purpose open-source LLM and GenAI models, such as Llama 3 and Mistral, using enterprise data. Databricks recommends a new approach for training LLM models with enterprise data called Retrieval Augmented Fine-tuning (RAFT), which combines both Retrieval-Augmented Generation (RAG) and model fine-tuning. —> Read more

👉 Release of Confluent Platform 7.7

Confluent announced the release of Confluent Platform 7.7, built on Apache Kafka 3.7. This update introduces significant features, including Confluent Platform for Apache Flink, a fully managed and serverless stream processing service (currently in Limited Availability), as well as a self-managed HTTP Source connector for ingesting data from external APIs. —> Read more

Introduction

Welcome to the Data Engineering Digest (DLD) newsletter series. This will be a periodic roundup of the latest and greatest in the world of data & data engineering in particular. We'll deliver a curated selection of finest news articles, blog posts, tutorials, discussions, and more within the data landscape.

✨ Featured

In a recent blog post I explored the difference between generic data lakehouses offered by some vendors, and open data lakehouses built on the foundation of open source tools and technologies. The Apache Hudi blog published a post providing an overview of the open data lakehouse architecture. It details the architecture layers, components, key technologies, advantages over previous data lake architectures, and use cases for implementing an open data lakehouse. —> Read more

📡 Open Source News

👉 Debezium Latest Official Release

Debezium 2.7.0.Final has been released. Some outstanding 140 issues have been fixed along with many new features and improvements in the core component as well as the stand-alone connectors. —> Read more

👉 Submission of OneTable to Apache Foundation

OneHouse news on submission of XTable (formerly known as OneTable) to the Apache Software Foundation Incubator was a major recent announcement. Top cloud vendors such as Microsoft and Google are already integrating XTable into their analytics platforms, Microsoft Fabric and BigLake respectively, to provide a unified logical lakehouse with interoperability between different open table formats. —> Read more

🛠 Practical Data Engineering

👉 Spark Repartition() Function

Spark pipelines often employ the df.repartition() function to optimise data processing, especially by consolidating small partitions before loading data into target storage. It's essential to remember that repartitioning in Spark is a unit of parallelism and data distribution used by the distributed compute engine, not necessarily a full bucketing or SQL-style group by operation. A blog post explains how Spark repartitioning works and what it actually does. --> Read more

👉 Apache XTable + Airflow

AWS published a blog post demonstrating how to use Apache XTable to convert open table format metadata to other formats. The blog post features a custom Airflow operator, XtableOperator(), designed for batch pipeline translations on the AWS platform. The operator's code is available on Github. This development suggests that unified open table format adoption is gaining some momentum. —> Read more

⚙️ Technical Deep Dive

👉 Apache Paimon’s Internal Design

While the 'big three' open table formats – Apache Hudi, Iceberg, and Delta Lake – dominate the market and discussions, Apache Paimon, a more recent “Flink table format”, has received less attention. If you're interested in learning more about Apache Paimon, a comprehensive blog post by Giannis delves into its design goals, internals, and key features. —> Read more

👉 How dlt Works Under the Hood

Discussions have been ongoing regarding the potential use of new open source data ingestion tools like Data Load Tool (dlt) as a replacement for more established ones like Airbyte in certain use cases (ex API data integration). A dlt project contributor has published a blog post detailing the internal data pipeline design and core functions of dlt, including data extraction, normalisation, and loading. The latest version leverages Apache Arrow's efficient in-memory data structure to optimise the entire pipeline. —> Read more

👉 Yet Another Kafka Explanation

There's a wealth of resources available explaining Kafka's architecture and internals. I found a recent blog post series on the topic, which provides clear and concise explanations of the concepts, accompanied by helpful visuals for those unfamiliar with Kafka's design and operation.

Kafka Architecture Overview | Design elements | Kafka Producer | Kafka Consumer

👉 DuckDB's Internal Memory and Buffer Management

If you've used DuckDB or are exploring its capabilities, you might wonder how it handles large datasets without memory limitations, a common issue with some Python dataframes like Pandas. DuckDB’s official blog has recently covered the engine's internal memory and buffer management. It explains how DuckDB leverages streaming execution to process queries without fully loading CSV or Parquet files into memory, and utilises disk spilling when intermediate results exceed memory capacity. —> Read more

👉 Snowflake's Micro Partitioning Internal Design

If you've been using Snowflake at your company, you're likely familiar with its internal partitioning feature called micro-partitioning. This process automatically divides tables into micro-partitions of 50 MB and 500 MB, organising data in a columnar format within each micro-partition. A concise and excellent blog post provides a clear explanation of micro-partitioning's internal design, complete with helpful visuals. —> Read more

👉 Overview of Kafka's Tiered Storage Design

Kafka 3.6, released in 2023, introduced a highly anticipated feature: Tiered Storage. This feature currently supports local and remote storage tiers, enabling the movement of inactive segments to a configurable deep storage solution like HDFS or S3 based on local retention settings. This provides a cost-effective and scalable way to retain historical data. Uber is credited with driving the tiered storage proposal [KIP-405], discussing the internals of the tiered storage architecture. —> Read more

🔎 Case Studies

👉 Implementation of dlt in Production

It's inspiring to hear about individuals and teams embracing new open-source tools and technologies in production environments. Data Load Tool (dlt) is a relatively new ETL library covered earlier, that has been adopted by some for production workloads. Alexander from Dataops explores the advantages and disadvantages of using dlt compared to more established data integration tools like Airbyte. —> Read more

👉 Notion's Data Architecture Evolution

Notion has unveiled their new data lakehouse architecture. They chose Apache Hudi as their table format due to its efficient incremental data ingestion capabilities, making it suitable for their update-heavy workloads. The architecture also incorporates event-based CDC ingestion using Debezium and Kafka. —> Read more

💬 Community Discussions

👉 This career advice from a senior data engineer highlights a key point in one of Reddit's highest-rated data engineering discussions in July:

While flashy tools and platforms come and go, a strong foundation in low-level skills like Bash, Git, SQL, pure Python development, and containerisation will take you much further. Juniors who prioritise these foundational skills before diving into advanced tools and stacks will be better-positioned for success.

🎥 Conferences & Events

👉 The annual virtual PrestoCon 2024 day organised by Linux Foundation/Presto Foundation took place in June, discussing topics like Presto 2.0 native C++ engine, and Presto usage at companies like Uber. A recap of the event and the main sessions is provided in this article. All the recorded 24 sessions can be found on Youtube.

In Part I we went over the origin and architecture of traditional table management systems and the first generation of Open Table Formats (OTF). In this final part, I will discuss the second and third generation OTFs.

2nd Generation OTF - The Rise of Log-oriented Table Format

Now that we have built a strong case for re-imagining and improving the open table format model, let's recap and list the major issues we identified with the previous generation of table formats:

• Tight coupling between physical partitioning and the logical partitioning scheme of the data.

• Heavy reliance on the file system or object store API for listing files and directories during the query planning phase.

• Relying on an external metadata store for maintaining table-level information such as schemas, partitions, and column-level statistics.

• Lack of support for record-level upsert, merge and delete.

• Lack of ACID and transactional properties.

Let's temporarily set aside the complexities of upsert and ACID transactions to focus on first three fundamental challenges. Given these constraints, we must consider how to decouple partitioning schemes from physical file layouts, minimise file system API calls for file and partition listings, and eliminate the reliance on an external metadata store.

To address these requirements, we need a data structure capable of efficiently storing metadata about data, partitions, and file listings. This structure must be fast, scalable, and self-contained, with no dependencies on external systems.

One solution to address these requirements is surprisingly simple, though not always the most obvious. Just as Jay Kreps and the engineering team at LinkedIn built Apache Kafka on the foundation of a simple append-only storage abstraction—an immutable log containing sequential records of events ordered by time— can we consider using a similar framework?

So the question is:

If immutable logs can store events representing facts that always remain true, effectively capturing the evolution of an application's state over time in systems like Apache Kafka, can't we apply the same basic principles to manage the state of table's metadata in our case?

By leveraging log files, we can treat all metadata modifications as immutable, sequentially ordered events. This aligns with the Event Sourcing data modeling paradigm, where we capture state changes at the partition and file level within transactional logs stored alongside the data.

Files and partitions become the unit of record for which the metadata layer tracks all the state changes in the log. In this design, the metadata logs are the first class citizens of the metadata layer.

Lets build a Simple Log-oriented Table

Let’s do a quick practical exercise to understand how we can design our new table format to capture and organise the metadata in log files.

In this exercise we will build a simple log-oriented metadata table format for capturing the filesystem and storage-level state changes such as adding and removing files and partitions, which can provide the event log primitives such as strong ordering, versioning, time travel and replaying event to rebuild the stage.

For capturing storage-level or file system state changes we need to consider two main file system objects, that is files and directories (i.e partitions) with following possible events:

Let’s assume a particular table contains three partitions in /year=/month=/day= format. In a most simple form, the metadata log can be implemented with following fields in an immutable log file:

timestamp|object|event type|value
20231015132000|partition|add|/year=2023/month=10/day=15
20231015132010|file|add|/year=2023/month=10/day=15/00001.parquet
20231015132011|file|add|/year=2023/month=10/day=15/00002.parquet
20231015132011|file|add|/year=2023/month=10/day=15/00003.parquet

Later if a file is removed, a new remove event can be captured at the end of the log file:

20231015132011|file|remove|/year=2023/month=10/day=15/00003.parquet

Managing Metadata Updates

Given the immutable nature of data lake storage systems like HDFS or object stores, metadata logs cannot be continuously appended to. Instead, each update resulting from data manipulation operations (e.g., new data ingestion) requires the creation of a new metadata file.

To maintain sequence and facilitate table state reconstruction, these metadata logs can be sequentially named and organised within a base metadata directory.

/mytable/
	/metadata_logs/
		000001.log
		000002.log
		000003.log
		000004.log

In order to rebuild the current table state, the order of the metadata logs in the metadata directory, and possibly the Timestamp field in the logs, can serve as the physical or logical clock which can provide strong sequential ordering semantics for replaying metadata events.

The query engines can scan the event log sequentially to replay all the metadata state change events in order to rebuild the current snapshot view of the table.

Log Compaction

Frequent data updates on large datasets can lead to a proliferation of metadata log files, as each change necessitates a new log entry.

Over time, the overhead of listing and processing these files during state reconstruction can become a performance bottleneck, negating the benefits of decoupling metadata management.

To mitigate this, a compaction process can merge individual log files into a consolidated file, removing obsolete records like superseded add and remove events. However, for time travel and rollback capabilities, these outdated events must be retained for a specified period.

By periodically executing background compaction jobs, we can generate snapshot logs encapsulating all essential state changes up to a specific point in time.

/mytable/
	/metadata_logs/
		000001.log
		000002.log
		000003.log
		000004.log
		snapshot_000004.log
		000005.log

In the above example, the snapshot log snapshot_000004.log has been generated for sequential log files 000001.log to 000004.log containing all the metadata transactions up to that point. To get the current table snapshot view, the latest snapshot file along with any additional new delta log files need to be scanned, which is now more optimised and efficient.

What did we just build?

We've successfully designed a foundational log-oriented table format that addresses our initial requirements by using simple, immutable transactional logs to manage table metadata alongside data files.

This approach serves as the bedrock for modern open table formats like Apache Hudi, Delta Lake, and Apache Iceberg.

Essentially:

This architectural shift marks a significant departure from previous table implementations by eliminating heavy reliance on the underlying storage system's metadata API, a potential performance bottleneck in large-scale data lakes.

By abstracting the physical file layout and tracking the table state (including partitions) at the file level within the metadata layer, these formats decouple logical and physical data organisation using the log-oriented metadata layer as shown below.

What about query performance?

In this architecture, the query performance is directly affected by how fast the required metadata can be retrieved and scanned during the query planning phase.

Using the underlying storage fast sequential I/O for reading metadata files provides much better performance that using their metadata APIs for gathering the required details such as the list of all sub-directories (partitions), files and retrieval of column-level statistics either from the footer section of the data files, or from the external metadata engine.

Is this a Novel Design?

The concept of using metadata files to track data files and associated metadata isn't entirely novel.

Key-value stores like RocksDB and LevelDB employ a similar approach, using manifest files to keep track of SSTables (data segments in LSM-Tree storage model) and their corresponding key ranges. These manifest files are cached in memory, enabling rapid identification of relevant SSTables without exhaustive directory scans suing the underlying storage APIs [1].

I wonder if those smart engineers behind the modern open table formats drew any inspirations from metadata management design in storage systems like RocksDB!

Adding Additional Feature

By adopting an event log and event sourcing model, we can readily implement additional valuable primitives:

• Event Replay - The ability to replay file and directory change event logs up to a specific version.

• Full State Rebuild - Compute engines can reconstruct the table's current state and identify active files and partitions by processing the metadata event log.

• Time Travel - Similar to event-based systems, we can revert to previous table versions using the event log and versioning mechanism.

• Event-Based Streaming Support - The transactional log inherently functions as a message queue, enabling the creation of streaming pipelines without relying on separate message buses.

Recall how Apache Hive manages column-level statistics (e.g., min/max values) for each table partition by storing records in a metadata database to optimize query performance. While binary formats like ORC and Parquet include file-level statistics, eliminating the need to query the Metastore, this approach still requires scanning and loading file footers during query planning, impacting scalability.

To address this, we can leverage our log-based metadata layer to store additional statistical metadata, optimising query performance by avoiding external system interactions and extensive file footer scans.

By consolidating file-level statistics into a small set of index files, we aim to reduce the I/O overhead associated with query planning from linear scaling (O(n)) to near-constant time (O(1)).

We could essentially follow the same metadata organisation, but use different naming conventions to manage column stats index. For each new data file loaded, a new delta index log can be generated to save the column stats records. When a compaction job runs to consolidate the metadata logs, it can also perform compaction on the column index logs to generate a snapshot file.

/mytable/
	/metadata_logs/
		stats_000001.log
		stats_000002.log
		stats_000003.log
		stats_000004.log
		stats_snapshot_000004.log
		stats_000005.log

Using a Consolidated Log File

By adopting a more structured file format capable of handling nested structures like JSON or Avro, we can optimise our design by consolidating all metadata within a single metadata file. This unified approach simplifies metadata management and reduces I/O overhead compared to managing multiple log files.

Furthermore, a single schema can be used to encapsulate different metadata types, streamlining the overall structure. To differentiate between metadata sets, we can employ a nested structure with distinct keys similar to following:

files:[
  {timestamp:20231015132000,type:partition,action:add,details:/.  data=20231015},
  {timestamp:20231015132000,type:file,action:add,details:/data=20231015/00001.parquet}
]

stats:[
  {timestamp:20231015132000,partition:/data=20231015,filename:00001.parquet,column_name:price,column_type:float,min:5,max:20},
  {timestamp:20231015132000,partition:/data=20231015,filename:00001.parquet,column_name:product,column_type:string,min:book,max:pen},
]

The above design is similar to how Apache Hudi manages the metadata in HFiles format. Another possible format is to use a single object for each data file containing all the related metadata using nested entries:

{
  "timestamp": 20231015132000,
  "type": "file",
  "action": "add",
  "details": "/data=20231015/00001.parquet"
  "column_stats":[
     {"column_name":"price","column_type":"float","min":5,"max":20},  {"column_name":"product","column_type":"string","min":"book","max":"pen"}
  ]
}

This is the format that Delta Lake uses by storing column-level statistics as a nested structure inside the main JSON transactional logs, under stats index.

Adding ACID Guarantees

A core design objective of open table formats is to enable ACID guarantees through the metadata layer. The new log-structured metadata approach inherently supports functions such as versioning and Snapshot Isolation via MVCC, addressing the previously discussed transaction isolation challenges in data lakes.

To provide Snapshot Isolation, writes can occur in following two steps:

1. Optimistically create or replace data files, or delete existing files on the underlying storage.

2. Atomically update the metadata transaction log with the newly added or removed files, generating a new metadata version.

This transactional mechanism prevents readers from encountering incomplete or corrupt data, a common issue in the previous table format generation, ensuring data integrity. By bypassing file system listing operations, we eliminate consistency issues like list-after-write on some object stores.

All three major table formats (Hudi, Delta Lake, Iceberg) implement MVCC with snapshot isolation to provide read-write isolation and versioning. They maintain multiple table versions as data changes, allowing readers to select files from the most recent consistent snapshot using the transaction log.

Multi-Write Concurrency can be facilitated through Optimistic Concurrency Control (OCC), which validates transactions before committing to detect potential conflicts. If concurrent writes target non-overlapping file sets, they can proceed independently.

However, if there's overlap, only one write succeeds while others are aborted during conflict resolution. All three major table formats employ some form of Optimistic Concurrency Control to manage concurrent writes and identify conflicts effectively.

The Origin of Modern Open Table Formats implementations

As previously discussed, the current generation of open table formats emerged to address the limitations of the previous generation of data management approaches on data lakes, and the foundation of these tools lies in the log-structured metadata organisation explored earlier.

• Apache Hudi, initiated by Uber in 2016, primarily aimed to enable scalable, incremental upserts and streaming ingestion into data lakes, while providing ACID guarantees on HDFS. Its design is heavily optimised for handling mutable data streams. The traditional snapshot and batch ingestion patterns used with Hive-style tables proved inadequate for low-latency use cases. An incremental approach that focused on new and updated data was necessary, but the immutability of HDFS posed challenges.

• Apache Iceberg originated at Netflix around 2017 in response to the scalability and transactional limitations of Hive's schema-centric, directory-oriented table format. The realisation that incremental improvements to Hive were insufficient drove the development of a new solution by changing the table design to instead track data in a table at the file level by pointing the table to an ordered list of files. Iceberg was born from this insight and employs a manifest-based metadata layer consisting of metadata, manifest list, and manifest files organised hierarchically.

• Delta Lake, introduced by Databricks in 2017 and open-sourced in 2019, emerged as the third major open table format. Its primary goal was to provide ACID transaction capabilities atop cloud object store-based data lakes. This was motivated by the absence of ACID guarantees, including cross-object consistency and query isolation, within cloud object stores.

• Apache Paimon is another notable and fairly recent open table format developed by the Apache Flink community in 2022, as the "Flink Table Store" and a lakehouse streaming storage layer with the main design goal of handling high throughput and low latency streaming data ingestion. However it has yet to gain any significant traction in comparison to the dominant trio.

These projects have significantly streamlined data management for users by automating optimisations, compaction, and indexing processes. This relieves data engineers from the burden of complex low-level physical data management tasks.

In Part 1 we questioned whether we could build a system that can combine the benefits of the traditional monolithic DBMS and disaggregated data lake systems. We can now declare that what we have is:

A powerful combination of the encapsulation and abstraction found in traditional DBMS physical layers with the openness, interoperability, and flexibility of modern open table formats.

Industry Adoption

The past few years have witnessed widespread adoption and integration of next-generation open table formats across various data tools and platforms.

All the major open table formats have gained traction and popularity while a fierce competition for market dominance has been going on mainly by the SaaS vendors providing these products as a managed service.

Major cloud providers have also embraced one or all of the big three formats, with Microsoft fully committing to Delta Lake for its latest OneLake and Microsoft Fabric analytics platforms, and Google adopting Iceberg as the primary table format for its BigLake platform. Cloudera, a leading Hadoop vendor, has also built its open data lakehouse solution around Apache Iceberg.

Prominent open source compute engines like Presto, Trino, Flink, and Spark now support reading and writing to these open table formats. Additionally, major MPP and cloud data warehouse vendors, including Snowflake, BigQuery, and Redshift, have incorporated support through external table features.

Beyond these tools and platforms, numerous companies have publicly documented their migration to open table formats.

3rd Generation OTF - Unified Open Table Format

The evolution of open table formats has marched on with a new trend since last year: cross-table interoperability.

This exciting development aims to create a unified and universal open table format that seamlessly works with all major existing formats under the hood.

Currently, converting between formats requires metadata translation and data file copying. However, since these formats share a foundation and often use Parquet as the default serialisation format, significant opportunities for interoperability exist.

A uniform metadata layer promises a unified approach for reading and writing data across all major open table formats. Different readers and writers would leverage this layer to interact with the desired format, eliminating the need for manual format-specific metadata conversion or data file duplication.

The State of Art

LinkedIn engineers pioneered one of the earliest attempts at a unified table API with OpenHouse introduced in 2022. Built on top of Apache Iceberg, OpenHouse offered a simplified interface for interacting with tables, regardless of their underlying format, through a RESTful Table Service seamlessly integrated with Spark.

While OpenHouse was a great effort, it lacked comprehensive interoperability and format conversion capabilities. Additionally, its open-sourcing in 2024 came relatively late compared to other emerging projects that had already gained significant traction, specially with giant tech companies such as Databricks, Microsoft and Google backing following projects.

Apache XTable (formerly known as OneTable), introduced by OneHouse in 2023, provides a lightweight abstraction layer for generating metadata for any supported format using common models for schemas, partitioning details, and column statistics. In terms of metadata layout, XTable stores metadata for each format side-by-side within the metadata layer.

XTable uses the latest snapshot of the primary table format, and generates additional metadata for target formats. Consumers can either use the primary format, or the target formats to read and write and get the same consistent view of the table’s data.

Databricks introduced Delta UniForm in 2023. Delta UniForm automatically generates metadata for Delta Lake and Iceberg tables while maintaining a single copy of shared Parquet data files. It's important to note that UniForm, primarily sponsored by Databricks, seems focused on using Delta Lake as the primary format while enabling external applications and query engines to read other formats.

How do they compare?

LinkedIn's OpenHouse project offers more of a control plain than a unified table format layer.

Comparing Apache XTable to Delta Uniform, XTable takes a broader approach, aiming for full interoperability and allowing users to mix and match read/write features from different formats regardless of the primary format chosen.

As an example, XTable could enable incremental data ingestion into a Hudi table (leveraging its efficiency) while allowing data to be read using Iceberg format by query engines like Trino, Snowflake, or BigQuery.

That being said, we're still in the early phases of development of uniform table format APIs. It will be exciting to see how they progress over the coming months.

Data Lakehouse

That brings us to the last part of this blog post to explore the concept of a data lakehouse without which our discussion would be incomplete. Let’s define what a data lakehouse stands for:

A data lakehouse represents a unified, next-generation data architecture that combines the cost-effectiveness, scalability, flexibility and openness of data lakes, with the performance, transactional guarantees and governance features typically associated with data warehouses.

That definition sounds very similar to what open table formats stand for! That’s because the lakehouse foundation is based on leveraging open table formats for implementing ACID, auditing, versioning, and indexing directly on low-cost cloud storage, to bridge the gap between these two traditionally distinct data management paradigms.

In essence, data lakehouse enables organisations to treat data lake storage as if it were a traditional data warehouse and vice versa. They offer the flexibility and decoupled architecture of data lakes—allowing for the storage of unstructured and semi-structured data in open formats and the use of diverse compute engines—combined with the performance, transactional capabilities, and full CRUD operations characteristic of data warehouses.

This vision was initially pursued by SQL-on-Hadoop tools to bring data warehousing to Hadoop platforms, but only getting fully realised recently with the advancements in the data landscape.

Non-Open vs Open Data Lakehouse

It's crucial to differentiate between a general "data lakehouse" and an "open data lakehouse".

While top cloud vendors like AWS and Google often label their data warehouse-centric platforms as data lakehouses, their definition is broader.

Their emphasis is on their data warehouses' ability to store semi-structured data, support external workloads like Spark, enable ML model training, and query open data files—all characteristics traditionally associated with data lakes. These platforms also typically feature decoupled storage and compute architectures.

It was around 2020-2021 Amazon started promoting a lake house concept comprised of Amazon Redshift data warehouse implemented over new RA3 managed storage, plus Redshift Spectrum, before the wider adoption of the "lakehouse" term by other vendors like Databricks and the data community in general, as a new approach to data warehousing.

Google has similarly promoted its analytics lakehouse architecture, outlined in a whitepaper published in 2023, providing a blueprint for building a unified analytics lakehouse using either BigQuery as the first choice, or open Apache Iceberg and BigLake platform.

On the other hand, open data lakehouse primarily leverage open table formats to manage data on low-cost data lake storage. This architecture promotes higher interoperability and flexibility, allowing organisations to select the optimal compute and processing engine for each job or workload.

By eliminating the need to duplicate and move data across systems, open data lakehouses ensure that all data remains in its original, open format, serving as a single source of truth.

Vendors such as Databricks, Microsoft OneLake, OneHouse, Dremio, and Cloudera have positioned themselves as providers of managed open data lakehouse platforms on cloud.

Conclusion

This post series has covered a lot of ground, taking you on a journey through the evolution of data.

I am personally always interested in understanding how a technology came to be, the major architectural changes and evolutions it underwent, and the design goals and motivations behind it.

I hope you have enjoyed the ride and now have a better understanding of where we are in the technology timeline and how we got here.

References

[1] Dong, S., Callaghan, M., Galanis, L., Borthakur, D., Savor, T., & Strum, M. (2017, January). Optimizing Space Amplification in RocksDB. In CIDR (Vol. 3, p. 3).

If you have been following trends in data engineering landscape over the past few years surely you have been hearing a lot about Open Table Formats and Data Lakehouse, if not already working with them! But what is all the hype about table formats if they have always existed and we have always been working with tables when dealing with structured data in any application?

In this blog post, we will delve into the history and evolution of open table formats within the data landscape. We will explore the challenges that led to their inception, the key innovations that have defined them, and the impact they have had on the industry.

By understanding the journey from traditional database management systems to the modern open table formats, we can better appreciate the current state of data technology and anticipate future trends.

In Part I, we will discuss the origin and history of storing and managing data in tabular format, and the emergence of first generation open table format.

In Part II, Second and third generation open table formats will be discussed.

The origin of Table Formats

Presenting information in a two-dimensional tabular format has been the most fundamental and universal method for displaying structured data, with roots tracing back over 3500 years to the old Babylonian period when the most ancient table data were recorded on clay tablets.

The modern concept of database tables emerged with the invention of relational databases, inspired by E.F. Codd's paper on the Relational Model published in 1970.

Since then, table formats have been the primary abstraction for managing and working with structured data in relational database management systems, such as the pioneering System R. Thus, the concept of table formats in storage systems is not novel, having been a staple for the past half-century.

Table Format Abstraction

Data tables are logical datasets, an abstraction layer over physical data files stored on disk, providing a unified, two-dimensional tabular view of records. The storage engine combines records from various objects for a dataset and presents them as one or more logical tables to the end user.

This logical table presentation offers the advantage of decoupling and hiding the physical characteristics of data from applications and users, allowing for the evolution, optimisation, and modification of physical implementation details without impacting users.

So, we've been hearing a ton about open table formats lately, but what's the big deal? And what's the difference between open and non-open or closed formats anyway? To figure that out, let's dive into how a general database management system is implemented.

Relational Table Format

Prior to the Big Data era and emerge of Apache Hadoop in mid 2000s, traditional Database Management Systems (DBMS) adhered to a monolithic architectural design.

This architecture is comprised of several highly interconnected and tightly coupled layers, each dedicated to specific functionalities essential for the database's operation but all the components are combined to form a single unified system. The storage layer, in particular, managed the physical aspects of data persistence.

At the core of this structure lays the Storage Engine. This component served as the lowest abstraction level, overseeing the physical organisation and management of data on disk. Critical tasks such as transaction management, concurrency control, index management, and recovery were also handled by the storage engine.

Crucially, the query and presentation layers operated in isolation from the intricacies of the storage layer. Data traversed these layers during read and write operations, with each layer imposing its own abstraction. This encapsulation meant that the physical layout and storage format of data remained concealed from external systems.

What’s the implication?

What that means is that, it was not possible to directly access or manipulate database's physical data files using other systems or programming languages like Python as we do now a days.

Moreover, it lacked interoperability as it one cannot just copy database files to another system or simply pointing a generic query engine at a the database's files on OS and interact with the data.

Given these constraints, the concept of an Open Table Format (OTF) as we understand it today was non-existent. Traditional databases employed proprietary storage formats tightly integrated with their specific implementations.

Are we saying there is a fundamental flaw in traditional DBMS architecture!?

Many would argue that this is actually not a bad design at all. After all, normal users shouldn't care, know or even change how the underlying physical layer is implemented because they can do catastrophic things! additionally, considerable technical expertise is applied to integrate interconnected components into a cohesive complex system.

Considering software design best practices it’s a valid argument as the design fully encapsulates the complexities of managing it, however the tradeoff is that this closed and tightly coupled design hinders interoperability, portability and open collaboration to build scalable and innovative systems based on open standards.

But is it possible to have the best of both worlds? That is, having highly interoperable, portable, open and scalable data systems and still be able to encapsulate the complexities of managing the low-level tasks such as managing data files on disk? We will find out later in the article.

Hadoop and Big Data Revolution

Let's fast forward from the 1970s to 2006, when the BIG DATA Revolution took place and the data landscape underwent a seismic shift when Apache Hadoop project was born out of Yahoo leading to disassembly of database systems.

I will not discuss the internals of Apache Hadoop and its architecture as there are lots of material available if you are unfamiliar with it. But one major architectural breakthrough was the decoupling of storage and compute.

This fundamental architectural change allowed for the storage of vast amounts of data in common semi-structured text-based formats such as CSV and JSON, or binary formats like Avro, Parquet, and ORC, on HDFS distributed files system deployed on affordable commodity hardware.

Data could be stored much like files on a local file system and processed using distributed processing frameworks of choice like MapReduce, Pig, Hive, Impala and Presto.

For the first time, businesses could store vast amounts of data in their preferred open formats, and leverage different compute engines for various workloads, enabling large-scale analytics. This was a game-changer for those accustomed to inflexible, expensive, monolithic storage systems and proprietary data warehouses.

But the real breakthrough, as stated by AMPLab co-director Michael Franklin was achieving data independence as result of the new decoupled architecture:

That is why Big Data was such a Big Hype at the time generating such excitement with enterprises rushing to bring the elephant in the room – a similar level of hype surrounds Generative AI today, creating a sense of déjà vu for some. Nevertheless, the big data was a true revolution, breaking free from the confines of traditional systems and providing the foundation of many innovations that followed next in the open data ecosystem.

1st Generation OTF - The Birth of Open Table Format

The initial release of Apache Hadoop presented significant challenges for data engineers.

Expressing data analysis and processing workloads in MapReduce logic using Java was both complex and time-consuming. Moreover, Hadoop lacked a mechanism for storing and managing schemas for datasets on its file system.

While engineers appreciated Hadoop's flexibility, they yearned for the familiarity of SQL and the two-dimensional table format inherent to relational databases.

To bridge this gap, Facebook (now Meta), an early and influential Hadoop adopter, initiated the Hive project. The goal was to introduce SQL and tabular structures, familiar from traditional relational databases, into the Hadoop and HDFS ecosystem.

However a key distinction was its new architectural approach:

Being built on top of the decoupled physical layer, leveraging open data formats stored on HDFS distributed file system.

Impact of Apache Hive

Facebook open-sourced Hive in 2008, making it available to the broader community. A few years later, Cloudera, a prominent Hadoop vendor, developed Apache Impala.

Similar to Hive, Impala offered table management on HDFS, incorporating schema management and features like automatic file format conversion and compaction.

The introduction of Apache Hive and Impala into the Hadoop stack, the concept of open table formats built upon open file formats was born. Managed and external tables, along with directory-based partitioning, became the primary abstractions for data ingestion, data modeling, and management within the Hadoop ecosystem.

This new data architecture enabled data integration and processing pipelines to operate independently, loading data files into HDFS in the appropriate format without requiring knowledge on how and by which query engine the data would be consumed.

Evolution of Columnar Binary File Formats

Another pivotal advancement was the development of efficient columnar open file formats. This began with RCFiles, a first-generation columnar binary serialisation framework from Apache Hive project.

Subsequent innovations included Apache ORC as an improved version of RCFile, released in 2013, and Apache Parquet, a joint effort between Twitter and Cloudera, also released in 2013.

These new open file formats dramatically enhanced the performance of OLAP-based analytical workloads on Hadoop, laying the groundwork for building OLAP storage engines directly on data lakes.

Since then, ORC and Parquet have become the de facto standard open file format for managing data at rest on data lakes, with Parquet being more popular and enjoying wider adoption and support in the ecosystem.

Next we will dive deeper into how Hive table format is structured, but before that let's generalise the physical design the engines such as Hive and Impala use which heavily relies on the file system directory hierarchy. Lets call it directory-oriented table formats.

Directory-oriented Table Formats

The most fundamental approach to treating data as a table in a distributed file system such as HDFS (i.e., a data lake) involves projecting a table onto a directory containing immutable data files and potentially sub-directories for partitioning.

The important factor to note is that this architecture is inherently tied to the physical file system layout, relying on file and directory operations for data management. This has been the standard practice for storing data in data lakes since the inception of Hadoop.

Directory-based partitioning allows for organising files based on attributes like event or process date. Schema information can be embedded within data files or managed externally by a schema registry.

Since table partitions are represented as sub-directories, it becomes the responsibility of the query engines to parse and scan each partition represented as a sub-directory in order to identify the relevant data files during query planning phase.

This implies that the physical partitioning is tightly coupled with the logical partitioning on the table level with its own constraints which will be discussed later.

Now that we have covered what a directory-based table looks like, lets look at the Hive table format.

Hive Table Format

With the presented storage model, it’s fair to say that Apache Hive is a directory-oriented table format, relying on the underlying file system's API for mapping files to tables and partitions. Consequently, Hive is heavily influenced by the physical layout of data within the distributed file system.

Hive employs its own partitioning scheme, using field names and values to create partition directories. It manages schema, partition, and other metadata in a relational database known as the Metastore.

The Significant shift so far with Hive + Hadoop is:

Unlike traditional monolithic databases, Hadoop and Hive's decoupled approach allows other query and processing engines to process the same data on HDFS using Hive engine’s metadata.

Following example shows a typical Hive temporal partitioning based on year, month and day.

This leads to another major difference between the new data architecture and traditional DBMS systems: While traditional systems tightly bind data and metadata like table definitions, the new paradigm separates these components.

This decoupling offers great flexibility. Data can be ingested into data lakes without accompanying metadata, and multiple processing systems can independently assign their own metadata or table definitions to the same data.

Moreover, a centralised schema registry (such as the Hive Metastore, which has become the de facto standard) allows any processing engine to interact with data in a structured tabular format, using familiar SQL or python languages using other computation frameworks such as Spark, Presto and Trino.

By accessing table metadata within the registry, query engines can determine file locations on the underlying storage layer, understand partitioning schemes, and execute their own read and write operations.

I hope you now understand why we refer to this design as open and perhaps begin to appreciate its flexibility and open architecture compared to previous generations of database systems.

Drawbacks and Limitations and the Directory-oriented and Hive table Format

For nearly a decade, from 2006 to 2016, Hive reigned supreme as the most popular table format on Hadoop platforms. Tech giants like Uber, Facebook, and Netflix heavily relied on Hive to manage their data.

However, as these companies scaled their data platforms, they encountered significant scalability and data management challenges that Hive couldn't adequately address.

Let's delve into the shortcomings of the directory-oriented table formats and Hive-style tables that prompted the engineers at this tech companies to seek alternatives.

First lets look at challenges and drawbacks of directory-oriented table format, the foundation upon which Hive has been developed:

• High Dependency on Underlying File System - This architecture heavily relies on the underlying storage system to provide essential guarantees like atomicity, concurrency control, and conflict resolution. File systems lacking these properties, such as Amazon S3's absence of atomic rename, necessitate custom workarounds.

• File Listing Performance - Directory and file listing operations can become performance bottlenecks, particularly when executing large-scale queries. Cloud object stores like S3 impose significant limitations on directory-style listing operations. Each LIST request returns a maximum of 1000 objects, necessitating multiple sequential requests, which can be slow due to latency and rate limiting. This significantly impacts performance when dealing with large datasets.

• Query Planning Overhead - On distributed file systems like HDFS, query planning can be time-consuming due to the need for exhaustive file and partition listing. This is especially pronounced when dealing with a large number of files and partitions.

Drawbacks and challenges of using Hive-style partitioning:

• Over Partitioning - Tightly coupling physical and logical partitioning can lead to over-partitioning, especially with high-cardinality partition columns like year/month/day. This results in excessive small files, increased metadata overhead, and slower query planning due to the need to scan numerous partitions. Over-partitioning is particularly detrimental to MPP engines like Hive, Spark, and Presto, as they struggle with query planning and scanning a large number of small partitions.

• Cloud Effect - Cloud data lakes exacerbate over-partitioning issues due to API call limitations. Jobs scanning many partitions and files often encounter throttling, leading to severe performance degradation.

• Too Many Small Files - Incorrect partitioning schemes can create numerous small files, having negative impact on different layers, slowing down queries and job planning and re-partitioning requires rewriting the entire dataset, a costly and time-consuming process.

• Poor Performance - Queries on Hive-style directory-based partitions can be slow without specifying the partition key for data skipping, especially with deep partition hierarchies. Accidental full table scans become common, leading to inefficient and lengthy query execution.

• Accidental Costly Queries - Accidental full table scans can result in launching large queries and jobs. During my years of managing a Hadoop platform, I had to explain many times to end-users why their simple Hive query was taking a long time to run due to scanning large number of partitions during query planning phase.

Imagine a Hive table being partitioned by 20 provinces, followed by year=/month=/day=/hour= partitions. Such a table would accumulate over 1 million partitions in 6 years.

Drawbacks of using External Metastore

In addition to the above drawbacks, the Hive-style table using an external Metastore add more challenges into mix:

• Performance Bottleneck - Both Hive and Impala rely on an external metadata store (typically a relational database like MySQL or PostgreSQL), which can become a performance bottleneck due to frequent communication for table operations.

• Metadata Performance Scalability - As data volumes and partition counts grow, the Metastore becomes increasingly burdened, leading to slow query planning, increased load, and potential out-of-memory errors. These issues have been extensively documented and addressed by the community. Many companies such as Airbnb have experience Metastore performance challenges before upgrading their platform.

• Single Point of Failure - The Metastore represents a single point of failure. Crashes or unavailability can cause widespread query failures. Implementing high availability is crucial to mitigate downtime.

• Network latency - Network latency between the query engine and the external Metastore, as well as the underlying relational database, can impact overall performance.

• Inefficient Statistics Management - Hive's reliance on partition-level column statistics, stored in the Metastore, can hinder performance over time. Wide tables with numerous columns and partitions accumulate vast amounts of statistical data, slowing down query planning and impacting DDL commands like table renaming.

A First-hand Experience

I have personally faced many of the above challenges working with Hive in production for many years. In a recent project our development team had to rename some large and wide managed Hive tables with about 10k partitions and the rename would just hang and not complete even after many hours.

After investigation I found that for each table there are about 300k statistical records stored which Hive is trying to gather details and update these records. Even after rebuilding the index on the stats table in PostgreSQL database, the issue didn't fully get resolved.

I believe I’ve made a pretty strong case against Hive table format and its underlying directory-oriented architecture. Apache Hive has served the big data community well for nearly a decade, but its time to improve and develop something more efficient and scalable.

Transactional Guarantees on Data Lakes

Before presenting the next evolution of table formats, let's also examine some common challenges associated with implementing database management systems on a data lake backed by distributed file systems such as HDFS or object stores like S3.

These challenges are not specific to Hive or any other data management tool but are generally related to the ACID and transactional properties of traditional DBMS systems.

• Lack of Atomicity - Writing multiple objects simultaneously within a transaction is not natively supported, hindering data integrity.

• Concurrency Control Challenges - Concurrent modifications to files within the same directory or partition can lead to data loss or corruption due to the absence of transaction coordination.

• Absence of Transactional Features - Data lakes build on HDFS or object stores lack built-in transaction isolation and concurrency control, requiring organisations to relax consistency requirements or implement custom solutions. Without transaction isolation, readers can encounter incomplete or corrupt data due to concurrent writes.

• For read and write isolation, downstream consumers would have to implement custom mechanisms to ensure data consistency by waiting for upstream batch data processing workload to complete before initiating their jobs.

• Support for Record-Level Mutations - The immutable nature of underlying storage systems prevents direct updates or deletes at the record level in data files.

• Object Store Challenges - Object stores like S3 historically lacked strong read-after-write consistency, prompting some organisations to use staging clusters (e.g., HDFS) as an intermediate step before final data placement. Additionally, the absence of atomic rename operations has posed challenges for distributed processing engines like Spark and Hive, which rely on temporary directories for data staging before finalising output.

Hive Transactional Tables

Hive ACID feature was the first attempt to introduce structured storage guarantees, particularly ACID transactions (Atomic, Consistent, Isolated, Durable), to the realm of immutable data lakes.

Released in Hive version 3 (2016), this feature marked a significant leap forward by providing stronger consistency guarantees like cross-partition atomicity and isolation. Additionally, it offered improved management of mutable data on data lakes through upsert functionality.

But addition of ACID to Hive didn’t solve the fundamental issues because:

Hive ACID tables remained rooted in the directory-oriented approach, relying on a separate metadata store for managing table-level information within the underlying data lake storage layer.

Several attempts were made to integrate Hive ACID into the broader data ecosystem. Hortonworks developed the Hive Warehouse Connector to enable Spark to read Hive transactional tables, initially relying on Hive LLAP component.

Cloudera later introduced Spark Direct Reader mode in 2020, allowing direct file system access without Hive LLAP dependency.

Despite these efforts, I would say Hive ACID didn't catch the imagination of the community as it failed to gain widespread adoption due to its underlying design limitations. Support for reading and writing Hive ACID tables remained inconsistent across the ecosystem, with many prominent tools like Presto offering limited or no support.

That’s the end of Part I. In Part II next generation open table formats will be discussed.

While the widespread hype surrounding Generative AI and ChatGPT took the tech world by storm, 2023 witnessed yet another exciting and vibrant year in the data engineering landscape, steadily grown more diverse and sophisticated, with continuous innovation and evolution across all tiers of the analytical hierarchy.

With the continued proliferation of open source tools, frameworks, and solutions, the options available to data engineers have multiplied! In such rapidly changing landscape, the importance of staying abreast of the latest technologies and trends cannot be overstated. The ability to choose the right tool for the right job is a crucial skill, ensuring efficiency and relevance in the face of evolving data engineering challenges.

Having closely followed data engineering trends in my role as a senior data engineer and consultant, I'd like to present the open source data engineering landscape at the beginning of 2024. This includes identifying key active projects and prominent tools, empowering readers to make informed decisions when navigating this dynamic technological landscape.

Why Present Another Landscape?

Why make the effort to present yet another data landscape!? There are similar periodic reports such as the famous MAD Landscape , State of Data Engineering and Reppoint Open Source Top 25, however the landscape I'm presenting is focused solely on open source tools mainly applicable to data platforms and data engineering lifecycle. The MAD Landscape provides a very comprehensive view of all tools and services for Machine Learning, AI and Data, including both commercial and open source, while the landscape presented here provides a more comprehensive view of active open source projects in the Data part of MAD. Other reports such as Reppoint Open Source Top 25 and Data50 focus more on the SaaS providers and startups, whereas this report focuses on the open source projects themselves, rather than the SaaS services.

Annual reports and surveys such as Github's state of open source , Stackoverflow annual survey and OSS Insight reports are also great sources for gaining insight into what's being used or trending in the community, but they only cover limited sections (such as databases and languages) of the overall data landscape.

Therefore due to my interest in open source data stacks, I've compiled the open source tools and services in data engineering ecosystem.

So without further due, here is the 2024 Open Source Data Engineering Ecosystem:

Tool Selection Criteria

The available open source projects for each category are obviously vast, making it impractical to include every tool and service in the picture. Therefore I have followed the following criteria for selecting the tools for each category:

• Any retired, archived and abandoned projects are excluded. Some notable retired projects are Apache Sqoop, Scribe and Apache Apex which might still be used in some production environments.

• Projects which have been completely inactive on Github over the past year, and are hardly mentioned in the community are excluded. Notable examples are Apache Pig and Apache Oozie projects.

• Projects which are still quite new and have not gained much traction in terms of Github stars, forks, as well as blog posts, show cases and mentions in the online communities, are excluded. However some promising projects such as OneTable which has made some notable traction and are implemented on the foundation of existing technologies are mentioned.

• Data Science, ML and AI tools are excluded, except for ML platform and infrastructure tools, as I'm only focusing on what's related to data engineering discipline.

• Different types of storage systems such as relational OLTP and embedded database systems are listed. This is because data engineering discipline involves dealing with many different internal and external storage systems used in applications and operational systems (BSS), even if they are not part of the analytics stack.

• The category names are chosen as generic as possible based on where the tool fits in the data stack. For storage systems, main database model and database workload (OLTP, OLAP) are used for grouping and labeling the systems, but for instance "Distributed SQL DBMS" are also referred to as HTAP or scalable SQL databases in the market.

• Some tools could belong to more than one category. VoltDB is both an in-memory database and distributed SQL DBMS. But I have tried to place them in the category by which they are mostly recognised in the market.

• For certain database systems, there may be a blurry line regarding the category they actually belong to. For example ByConity claims to be a data warehousing solution, but is built on top of ClickHouse which is recognised as a Real-time OLAP engine. Therefore it is still unclear whether it is real-time (ability to support sub-second queries) OLAP system or not.

• Not all the listed projects are fully Portable open source tools. Some of the projects are rather Open Core than open source. In open core models, not all components of the full system, as offered by the main SaaS provider, are made open source. Therefore, when deciding to adopt an open-source tool, it is important to consider how portable and truly open source the project is.

Overview of Tool Categories

In the following section each category is briefly discussed.

1. Storage Systems

Storage systems are the largest category in the presented landscape, primarily due to the recent surge of specialized database systems. Two latest trending categories are vector and streaming databases. Materialize and RaisingWave are examples of open-source streaming database systems. Vector databases are also experiencing rapid growth in the storage systems field. I have placed vector storage systems in the ML Platform section since they are primarily used in ML and AI stacks. Distributed file systems and object stores are also placed their own related category, that is Data Lake Platform.

As mentioned in the selection criteria section, storage systems are grouped and labeled based on the main database model and workload. At the highest level, storage systems can be classified into three main classes: OLTP, OLAP, and HTAP. They can be further categorized based on SQL vs NoSQL for OLTP engines, and Offline (non-real-time) vs Real-time (sub-seconds result) for OLAP engines, as shown in the following figure.

2. Data Lake Platform

Data Lake platform has continued to mature in the past year, and Gartner has placed Data Lake in the slope of enlightenment in its 2023 edition of Hype Cycle for Data Management.

Source: Gartner

For storage layer, distributed file systems and object stores are still the main technologies serving as the bedrock for both on-premise and cloud-based data lake implementations. While HDFS is still the primary technology used for on-premise Hadoop clusters, Apache Ozone distributed object store is catching up to provide an alternative on-premise data lake storage technology. Cloudera, the main commercial Hadoop provider, is now offering Ozone as part of its CDP Private Cloud offering.

The choice of data serialization format impacts storage efficiency and processing performance. Apache ORC remains the preferred choice for columnar storage within Hadoop ecosystems, while Apache Parquet has emerged as the de-facto standard for data serialization in modern Data Lakes. Its popularity stems from its compact size, efficient compression, and wide compatibility with various processing engines.

Another key trend in 2023 is the decoupling of storage and compute layers. Many storage systems now offer integration with cloud-based object storage solutions like S3, leveraging their inherent efficiency and elasticity. This approach allows data processing resources to scale independently from storage, leading to cost savings and enhanced scalability. Cockroachdb supporting S3 as storage backend, and Confluent's offering of long-term Kafka topic data retention on S3 further exemplifies this trend, highlighting the growing use of data lakes as cost-effective, long-term storage solutions.

One of the hottest developments in 2023 was the rise of open table formats. These frameworks essentially act as a table abstraction and virtual data management layer sitting atop your data lake storage and data layer as depicted in the following diagram.

The open table format space is currently dominated by a fierce battle for supremacy between the following three major contenders:

• Apache Hudi: Initially developed and open-sourced by Uber, with main design goal for near-real-time data updates and ACID transactions.

• Apache Iceberg: Born from Netflix's engineering team.

• Delta Lake: Created and open-sourced by Databricks, with seamless integration with the Databricks platform.

The funding received by the leading SaaS providers in this space in 2023 – Databricks, Tabular, and OneHouse – emphasises market interest and their potential to further advance data management on data lakes.

Moreover, a new trend is now unfolding with the emergence of unified data lakehouse layers. OneTable (recently open-sourced by OneHouse) and UniForm (currently non-open source offering from Databricks) are the first two projects which were announced last year. These tools go beyond individual table formats, offering the ability to work with all three major contenders under a single umbrella. This empowers users to embrace a universal format while exposing data to processing engines in their preferred formats, leading to increased flexibility and agility.

3. Data Integration

The data integration landscape in 2023 witnessed not only continued dominance from established players like Apache Nifi, Airbyte, and Meltano, but also the emergence of promising tools like Apache Inlong and Apache SeaTunnel offering compelling alternatives with their unique strengths.

Meanwhile, Streaming CDC (Change Data Capture) has further matured, fueled by active development in the Kafka ecosystem. Kafka Connect and Debezium plugins have become go-to choices for near real-time data capture from database systems, while Flink CDC Connectors are gaining traction for deployments using Flink as the primary stream processing engine.

Beyond traditional databases, tools like CloudQuery and Streampipe are simplifying data integration from APIs, providing convenient solutions for ingesting data from diverse sources. which reflects the growing importance of flexible integration with cloud-based services.

In the realm of event and messaging middlewares, Apache Kafka maintains its strong position, though challengers like Redpanda are closing the gap. Redpanda's $100 million Series C funding in 2023 shows the growing interest in alternative message brokers offering low latency and high throughput.

4. Data Processing & Computation

The world of Stream processing continued to heat up in 2023! Apache Spark and Apache Flink remain the reigning champions, however Apache Flink made some serious headlines in 2023. Cloud giants like AWS and Alibaba jumping on board with Flink-as-a-service offerings, and Confluent's acquisition of Immerok for its own fully managed Flink as a service offering, show the momentum behind this powerful engine.

In the Python ecosystem, data processing libraries such as Vaex, Dask, polars, and Ray are available for exploiting multi-core processors. These parallel execution libraries further unlock possibilities for analysing massive datasets within the familiar Python environment.

5. Workflow Management & DataOps

The workflow orchestration landscape is arguably the most packed category in presented data ecosystem, filled with established heavyweights and exciting newcomers.

Veteran tools such as Apache Airflow and Dagster are still going strong and remains a widely used engines amid the recent hot debates in the community on unbundling, rebundling and bundling vs unbundling of workflow orchestration engines. On the other hand In the past two years, GitHub has witnessed the rise of several compelling contenders, capturing significant traction. Kestra, Temporal, Mage, and Windmill are all worth watching, each offering unique strengths. Whether focusing on serverless orchestration like Temporal, or distributed task execution like Mage, these newcomers can cater to the evolving needs of modern data pipelines.

6. Data Infrastructure & Monitoring

The recent Grafana Labs Survey, confirms Grafana, Prometheus and ELK stack continue to dominate the observability and monitoring landscape. Grafana Labs itself has been quite active, introducing new open-source tools like Loki (for log aggregation) and Mimir (for long-term Prometheus storage) to further strengthen its platform.

One area where open-source tools seem less prevalent is cluster management and monitoring. This likely stems from the cloud migration trend, reducing the need for managing large on-premise data platforms. While the Apache Ambari project, once popular for managing Hadoop clusters, was practically abandoned after the Hortonworks-Cloudera merger in 2019, a recent revival sparks some hope for its future. However, its long-term fate remains uncertain.

As for resource scheduling and workload deployment, Kubernetes seems to be the preferred resource scheduling specially on cloud-based platforms.

7. ML Platform

Machine Learning Platform has been one of the most active categories with unprecedented rise and interest in Vector databases, specialised systems optimised for the storage and retrieval of high-dimensional data. As highlighted by DB-Engines' 2023 report, vector databases emerged as the most popular database category in the past year.

MLOps tools also play an increasingly vital role in scaling ML projects efficiently, ensuring smooth operations and ML application lifecycle management. As the complexity and scale of ML deployments continue to grow, MLOps tools have become indispensable for streamlining development, deployment, and monitoring of ML models.

8. Metadata Management

In recent years, metadata management has taken center stage, propelled by the growing need to govern and improve management and access to data. However, the lack of comprehensive metadata management platforms prompted tech giants like Netflix, Lyft, Airbnb, Twitter, LinkedIn and Paypal to build their own solutions.

These efforts yielded some remarkable contributions to the open-source community. Tools like Amundsen (from Lyft), DataHub (from LinkedIn), and Marquez (from WeWork) are homegrown solutions, which have been open sourced and are under active development and contribution.

When it comes to schema management, the landscape remains somewhat stagnant. Hive Metastore continues to be the go-to solution for many as there are currently no alternative open source solution to replace it.

9. Analytics & Visualisation

In the Business Intelligence (BI) and visualisation domain, Apache Superset stands out as the most active and popular open-source alternative to licensed SaaS BI solutions.

As for distributed and Massive Parallel Processing (MPP) engines, some experts argue that big data is dead and majority companies don't require large-scale distributed processing, opting for single, powerful servers to handle their data volumes.

Despite this claim, distributed Massively Parallel Processing (MPP) engines like Apache Hive, Impala, Presto and Trino remain prevalent within large data platforms, especially for petabyte-scale data.

Beyond traditional MPP engines, uniform execution engines are another trend gaining traction. Engines such as Apache Linkis, Alluxio and Cube provide a query and computation middleware between upper applications and underlying engines.