✨ Featured - Table Buckets as a New Lakehouse Abstraction

Since AWS announced S3 Tables last December, many industry experts have shared their opinions through blogs and social media. Most of the feedback I've read so far has been somewhat negative. Critics like Daniel argue that S3 Tables create vendor lock-in. They see it as a proprietary solution that limits flexibility and openness required by true Open Lakehouse architectures.

Critics also highlight that S3 Tables mainly integrate with AWS services such as Glue, Athena, and EMR, lacking sufficient support for general-purpose tools and services.

Despite these criticisms, broader support for S3 Tables is emerging. PyIceberg now supports accessing S3 Tables through Glue Catalog, and preview support for querying S3 Tables using Iceberg REST API endpoints, such as Amazon SageMaker Lakehouse REST endpoints, has been added to DuckDB.

In this context, Werner Vogels, Amazon's CTO, recently published another excellent article on his All Things Distributed blog. He reflects on the 19-year evolution of Amazon S3, from simple object storage to a comprehensive data solution. Vogels emphasises how customer feedback has shaped the development of S3's major capabilities.

Vogels further explains that S3's latest innovation, S3 Tables, directly addresses customer feedback and pain points experienced working with open table formats on S3 object store. Its main goal is to enhance lakehouse data management by resolving these challenges. Vogels also discusses the ongoing tension between simplicity and velocity, a common trade-off faced in product development. I highly recommend reading his article.

Read the full article here

📡 Open Source News

👉 Apache Flink 2.0.0 Release

Apache Flink 2.0.0 has been officially released, marking the first major Flink release since version 1.x launched nine years ago. Key innovations include disaggregated state management, materialised tables for unified stream-batch processing, and enhanced SQL capabilities. Check out the full highlights of new features and improvements. --> Read More

👉 Apache Kafka 4.0 Release

Apache Kafka 4.0 is officially out, marking a significant milestone by removing the dependency on Apache ZooKeeper. Key features highlighted by the Confluent blog include a new consumer group protocol for improved rebalance performance, the introduction of Queues for Kafka to support traditional queue semantics, updated Java version requirements, and various enhancements through Kafka Improvement Proposals (KIPs). --> Read More

👉 DuckDB Now Has Its Own Web UI!

From DuckDB v1.2.1 a local web user interface (UI) for DuckDB, has been developed in collaboration with MotherDuck, aimed at enhancing user experience by simplifying database interactions. Using the new UI you can now execute SQL queries through interactive notebooks, explore databases and columns with advanced features like syntax highlighting and autocomplete. --> Read More

🛠 Practical Data Engineering

👉 Implementation of Medallion Architecture Using ClickHouse

A great hands-on example of implementing a medallion architecture using ClickHouse to process and analyse free data from the Bluesky social network. It highlights common challenges such as data duplication, malformed JSON, and inconsistent structures. It also demonstrates how ClickHouse features—including the JSON data type and the ReplacingMergeTree engine—effectively address these issues. --> Read More

👉 DuckDB’s Enhanced Lakehouse Analytics

Querying external Delta Lake tables from DuckDB has become significantly easier with recent improvements in the Delta extension. You can now attach to Delta tables and query them using aliases in a simpler and cleaner manner. Additionally, data-skipping enhancements accelerate federated queries over open table formats, demonstrating DuckDB's commitment to advance federated analytics capabilities over lakehouse table formats. --> Read More

👉 21 Reasons to Consider Apache Hudi!

While Apache Iceberg has emerged as the leading open table format and continues strong into 2025, Apache Hudi is making its case by highlighting 21 unique reasons to consider Hudi over Iceberg and Delta Lake. If you're currently evaluating open table formats for your next project or company, this comparison is worth checking out. --> Read More

⚙️ Technical Deep Dive

👉 Parquet Data Skipping Mechanisms

This concise article provides an excellent overview of how pruning and data skipping are typically performed on Parquet files using metadata and statistics at various levels, including row groups and pages. The described approach aligns with DataFusion's implementation of the Parquet reading and pruning pipeline, but the general design principles apply broadly to other Parquet readers as well. --> Read More

👉 What's the Difference Between Arrow Flight, ADBC, and Arrow IPC?

Still confused about the differences between Arrow Flight Protocol, ADBC, and Arrow IPC? This article clearly explains these key technologies within the Apache Arrow ecosystem. It also highlights the advantages of using Apache Arrow as a data interchange format, demonstrating how it enables faster and more efficient data exchange compared to traditional methods. --> Read More

👉 Airflow Data Intervals: A Deep Dive

Understanding Apache Airflow's concept of time and intervals—such as start time, execution time, and logical dates—can be challenging. This article dives deep into the significance of data intervals in Airflow, clearly explaining their critical role in effectively scheduling and executing workflows. It illustrates how data intervals ensure each DAG run processes complete and accurate datasets, thus promoting idempotency and enabling reliable backfilling of historical data. --> Read More

📃 Academic Papers

👉 OLAP DBMS Archetype For The Next Ten Years?

An insightful paper co-authored by Michael Stonebraker and Andy Pavlo provides a comprehensive overview of the evolution of database management systems (DBMSs) since 2005. The authors predict the lakehouse architecture as the "OLAP DBMS archetype" for the coming decade, offering a unified infrastructure capable of supporting both SQL and non-SQL workloads—a vision initially conceived but not fully realised during the Hadoop era. --> Read More

💼 Data Engineering Career

👉 How Will AI Disrupt Data Engineering?

This is one of the most compelling questions facing software and data engineers today. Tristan Handy, founder and CEO of dbt Labs, shares insightful perspectives on how artificial intelligence will impact data engineering roles and what the future might hold for data engineers. --> Read More

💪 Skill Up

👉 Learning ClickHouse Fundamentals

ClickHouse is offering free 3-hour training sessions on ClickHouse fundamentals scheduled for April 22 and May 15. If you're interested, be sure to sign up! --> Read More

📣 Vendors News & Announcements

👉 General Availability of Confluent's TableFlow

Confluent has announced the general availability of TableFlow, a technology that embraces stream-table duality. TableFlow simplifies the ingestion of event data from Kafka topics into structured lakehouse tables (currently supporting Iceberg and Delta Lake), abstracting away complex data engineering tasks. It manages the entire ingestion lifecycle, including schema evolution and table maintenance on Confluent Platform. --> Read More

👉 Materialize Self-Managed & Free Community Edition

The Materialize streaming database service provider has introduced two new offerings: a Self-Managed version and a Free Community Edition. The Self-Managed option allows deployment of Materialize in your own environment, providing greater control over performance and compliance. The Community Edition grants free access to Materialize's powerful features with specific usage limits, making it easier to test or run small-scale production workloads. --> Read More

🔎 Case Studies

👉 Streaming Data Ingestion Into Cloud Data Warehouse

This insightful case study by Canva compares different data ingestion approaches into cloud data warehouses. It contrasts their previous micro-batch, file-based ingestion method (using AWS services like Firehose) with a new architecture using Snowflake's managed Snowpipe Ingestion service. The new solution achieves low-latency ingestion of billions of events daily, significantly reducing query latency to under 10 minutes and lowering overall cloud costs. --> Read More

👉 Using Key-Value Stores for Exactly-Once Streaming Ingestion

Event deduplication and exactly-once processing guarantees are crucial challenges in building reliable streaming pipelines, often complicated by pipeline failures and network issues. One popular solution involves using an external, fast key-value store to track and eliminate duplicate events during streaming. MyHeritage shares their experience implementing exactly-once processing using Spark Structured Streaming and a key-value store for deduplication. --> Read More

🎥 Conferences & Events

👉 Google Cloud Next '25 Keynotes

Google Cloud Next '25 took place earlier this month, and both the opening keynote and developer keynote are now available on YouTube. The conference again focused heavily on AI, highlighting Google's Gemini LLM foundation model, AI agents, the Agent Development Kit, and new features in Vertex AI. On the analytics front, the new capabilities of Google's Data Science Agent look particularly exciting. Be sure to check out these keynotes to stay updated on the latest trends and technologies from Google Cloud.

Opening Keynote:

Developer Keynote:

Follow Me on LinkedIn

✨ Featured - The Rise of Single-Node Engines

In a recent blog post, Jordan Tigani, MotherDuck co-founder and former tech lead at Google BigQuery, highlighted that most companies don't actually deal with "Big Data." An analysis of half a billion sample queries run on Amazon Redshift revealed that over 80% of the queries processed less than 1 TB of data.

Even among the small percentage of businesses that do handle Big Data, the majority of queries (95%) are executed on smaller tables. In cases where companies have large datasets, with some tables exceeding 10 TB, about 96% of the queries still target smaller, likely aggregated tables of 100 GB or less, rather than the actual large tables.

A published paper by AWS also notes that most tables contain fewer than a million rows, with the vast majority (98%) having fewer than a billion rows.

These findings, combined with advancements in software and hardware technology that have significantly enhanced the processing capabilities of single-node systems, suggest that powerful emerging single-node compute engines like DuckDB could increasingly handle many non-big data use-cases. This would reduce the need for distributed processing frameworks like Spark, provided that the maturity and ecosystem support of these engines continue to evolve.

In a recent LinkedIn post, I shared a graph with some of these points, which sparked an engaging discussion among the data engineering community. The conversation featured a mix of opinions on the future of computing, and it's worth checking out.

💡Opinion

👉 The Infamous Rise of Notebook Engineers!

Daniel Beach has penned an intriguing article critiquing the growing use of notebooks in data engineering. He argues that many engineers misuse notebooks due to either a lack of technical skills or encouragement from vendors, like Databricks, promoting questionable practices. While notebooks are valuable for data analysts and scientists engaged in iterative data analysis, Daniel contends they are ill-suited for robust data engineering lifecycles. This misuse leads to poor coding standards, insufficient testing, and inadequate deployment practices. —> Read More

👉 The Analytics Personas in Business

Tristan Handy, the founder and CEO of dbt Labs, wrote an insightful piece about the key analytics personas in business. He critiques the common approach of treating the analytical process like an assembly line, which often fails to deliver significant insights and ROI. Handy suggests that the personas—primarily engineers, analysts, and decision-makers—should be seen as interchangeable "hats" that team members can wear when needed, while still maintaining their primary roles. This flexible approach fosters collaboration and enhances the overall effectiveness of the analytics process. —> Read More

📡 Open Source News

👉 Apache Airflow 2.10 Release

Apache Airflow 2.10 was released recently, introducing exciting new features such as support for multiple executors within a single Airflow environment. This allows users to assign different executors, like LocalExecutor and CeleryExecutor, to individual DAGs and even specific tasks. There are also numerous enhancements to Datasets and the UI, which are worth exploring. —> Read More

👉 Development of a New PostgreSQL DuckDB Extension

MotherDuck announced pg_duckdb, an open-source Postgres extension that embeds the DuckDB engine into the Postgres database for running analytical queries on Postgres data. This is a significant step towards easily transforming a popular OLTP system into an HTAP system using an embedded OLAP engine. The development looks promising, with multiple companies such as Microsoft, Neon, and Hydra joining the effort. The beta version has been released recently. —> Read More

👉 Ibis's Default Backend Change

The Ibis DataFrame library project announced that it will drop the Pandas and Dask backends in favour of making DuckDB its default backend. This decision is due to DuckDB's ease of installation, impressive speed, and strong support within the Python ecosystem. —> Read More

🛠 Practical Data Engineering

👉 Apache Airflow Alerting Techniques

The Google Data Analytics blog has provided a comprehensive overview of the alerting hierarchy in Apache Airflow, ranging from the top DAG level down to the individual task instance level. It details various alerting mechanisms that can be used to monitor the state of DAG runs and receive notifications about potential failures. The alerting techniques discussed are applicable not only to Google's Cloud Composer managed Airflow service but also to other Airflow deployments. —> Read More

👉 Best Practices for Optimising Airflow

The AWS blog has covered comprehensive strategies for optimising cost and performance in its Apache Airflow managed service, Amazon MWAA. Right-sizing remains crucial for achieving a balanced price-performance ratio in managed services. Amazon MWAA also supports auto-scaling, which can aid in this optimisation. The blog offers additional useful techniques for optimising DAG code to ensure that DAGs remain healthy, efficient, and scalable. These techniques can be applied to any Airflow deployment setup. —> Read More

Speaking of Airflow…

⚙️ Technical Deep Dive

👉 History and Evolution of Block Storage Services at AWS

Another fascinating story on the All Things Distributed blog explores the evolution of block storage services offered by Amazon Web Services (AWS). Written by one of AWS's leading engineers, it highlights key milestones in the development of Elastic Block Store (EBS), showcasing enhancements in performance, scalability, and continuous innovation. —> Read More

👉 The Internals of Apache Parquet

Vu has authored an insightful article on the internals of Parquet, the most popular cloud file format for data lakes. For those working with data lakes, understanding the design and architecture of common serialisation formats is invaluable for optimising storage and queries. —> Read More

👉 The Future of Distributed Systems and Their Storage Backend

A great article by Colin Breck on the future of distributed systems and highlighting current challenges, and major trends such as acceleration of object store adoption as the main storage backend abstraction for many analytical and transactional database systems. —> Read More

💪 Skill Up

In a recent LinkedIn post, I shared my top book recommendations for learning data engineering fundamentals. The feedback was overwhelmingly positive, with many comments emphasising the importance of selecting a good book and focusing on mastering the fundamentals. I have personally read all these books in recent years and have gained a lot from them.

👉 DataCamp’s Free Week

DataCamp is offering free access to its entire platform and all courses for a week, from November 4 to 10. This is a great opportunity to explore their courses and enhance your skills in the coming week! —> Read More

🔎 Case Studies

👉 Cost-effective Data Analytics Using Deterministic Sampling

Meta has shared valuable insights into its approach for achieving cost-effective data analytics through the use of deterministic sampling. This strategy is designed to balance the cost versus value trade-off, especially as data volumes and computation costs continue to rise exponentially. By employing deterministic sampling, Meta aims to reduce the overall cost and complexity of analytics without compromising the quality of insights. —> Read More

👉 Uber's New Declarative Batch ETL Framework

Uber has developed a modular declarative batch ETL framework called Sparkle, which leverages Apache Spark as the compute engine. Sparkle simplifies and standardises ETL pipeline development by allowing users to focus on expressing business logic as a sequence of transformation modules in SQL or Java/Scala/Python. It includes embedded unit testing and marks Uber's transition of all its batch ETL pipelines from Hive to Spark in 2023. —> Read more

👉 Self-service Kafka Platform Development Journey

Doordash has shared their journey in developing a self-service Kafka platform, aimed at addressing the challenges of managing Kafka infrastructure efficiently. This initiative was driven by the need to simplify the management of Kafka topics and resources, which was previously hindered by the use of low-level configuration management tools like Terraform. —> Read more

📣 Vendors News & Announcements

👉 Continuous Queries on Data Warehouse Systems

Google BigQuery has introduced a significant new feature called BigQuery continuous queries, currently available in Preview. This feature transforms BigQuery from a batch system into an event-driven streaming pipeline, leveraging the concept of Stream-Table Duality. It allows for the continuous ingestion of new events as data is loaded into BigQuery, enabling use cases like event-driven data processing, continuous record replication to a pub/sub queue or other streaming storage systems, real-time ML model integration, and Reverse ETL use cases. —> Read More

The Confluent blog has also published an article on leveraging this feature to stream data from BigQuery into the Confluent platform.

👉 New Google Managed Apache Kafka Service

Google has also announced a new managed service, Google Cloud Managed Service for Apache Kafka. This service abstracts the complexities of deploying and managing a Kafka cluster, offering features like security management, full management of brokers and storage, and automatic horizontal and vertical scaling. It also includes automatic storage tiering and lifecycle management to offload cold data to unlimited cloud storage. —> Read More

👉 Introduction of Conditional Writes on AWS S3

AWS recently announced the introduction of "Conditional Writes" on S3, marking a significant advancement in enhancing the reliability and efficiency of data operations, especially for distributed applications. This feature ensures that writes occur only if certain conditions are met, reducing the risk of unintentional data overwrites. It allows multiple clients to read and write to the same object without conflicts or concerns about overwriting each other's data. —> Read More

🎥 Conferences & Events

👉 Thinking Like an Architect

Gregor Hohpe delivered an insightful presentation titled "Thinking Like an Architect" at QCon London 2024. If you're interested in or work with data architecture, his talk provides valuable insights worth exploring. —> Watch

👉 Carnegie Mellon University's Intro to Database Systems Course - Fall 2024

The Fall 2024 session of Carnegie Mellon University's renowned "Intro to Database Systems" course was commenced in August. You can follow along with the course through recorded lectures available on their YouTube channel.

✨ Featured - The New Open Catalog War

In July this year, Snowflake open-sourced its Polaris Catalog under the Apache 2.0 license, with plans to submit it to the Apache Incubator program. Polaris is a catalog service designed for Apache Iceberg but can extend to other major open formats as well.

The question is: why does Apache Iceberg, which already has its own metadata layer, need a separate catalog service?

While Delta Lake, Apache Hudi, and Apache Iceberg open table formats provide their own metadata, each query engine (such as Spark, Flink, Presto or Trino) must perform separate integrations for tasks like schema discovery and data operations.

An open unified catalog service like Polaris simplifies this by streamlining multi-engine interoperability. It also offers enhanced features like improved search, data discovery, tagging, and governance, including access control through a unified, open, and vendor-agnostic interface compatible with various storage engines.

Polaris has the potential to become the standard catalog service for data lakehouse platforms, much like Hive Metastore was for Hadoop-based systems. Currently, catalog options include proprietary tools like AWS Glue Catalog and Databricks’ Unity Catalog, which was also open-sourced in June 2024.

Snowflake’s decision to open-source Polaris may have been influenced by Databricks' move to open source Unity Catalog. While some may see this as a new “catalog war” driven by marketing strategies, as noted by Chris, there’s still hope that these moves will lead to production-ready, open-source catalog services that can finally provide an alternative solution to Hive Metastore after all these years.

📡 Open Source News

👉 Seamless Integration of dbt and Airflow

Building and scheduling data pipelines using dbt models and workflow orchestration tools like Airflow has become a standard practice in data engineering for running transformation workflows. In response to the growing demand for seamless integration between the two systems, Astronomer developed a Python package called Cosmos. This package simplifies running dbt models within Airflow DAGs through a new Airflow task type, DbtDag. The latest version, 1.5.1, was released on July 17, so if you're using dbt and Airflow, it's definitely worth checking out. —> Read More

👉 What's New in Apache Kafka 3.8.0

The release of Apache Kafka 3.8.0 was recently announced, bringing several new features and improvements. This post on the official Apache Kafka site provides an overview of key updates, including support for compression levels, a new consumer rebalance protocol, and re-bootstrapping capabilities. —> Read More

👉 Apache Flink 1.20: New Features and the Road to Flink 2.0

Apache Flink 1.20 was also recently released, and the Confluent blog highlights the major improvements and features in this update. Notable enhancements include improvements to the bucketing feature for Flink SQL tables, allowing users to specify the number of buckets in the DISTRIBUTED BY clause, and the introduction of Flink SQL Materialised tables, which are automatically refreshed in the background as data streams in. Additionally, there are various operational improvements. According to reports, this may be the last minor release before Flink 2.0. —> Read More

🛠 Practical Data Engineering

👉 Mastering Airflow Trigger Rules

Astronomer, a managed Airflow service, has provided an overview of Airflow trigger rules with a visual guide to help new Airflow developers better understand and apply the right trigger rules in their DAGs. For new engineers, grasping all the trigger rules can be challenging, but it's a crucial aspect for effectively managing dependencies between upstream and downstream tasks. —> Read More

👉 A Radical Simplicity Approach to Data Engineering

The tds blog recently shared a great post about the trade-off between simplicity and functionality in software projects, including data engineering. The author advocates for a philosophy of "Radical Simplicity," where simple, straightforward solutions are prioritised over complex ones. This resonates with me, as I believe complexity should only be introduced when absolutely necessary. —> Read More

👉 Hands-On Guide: Installing and Integrating Polaris OSS

If you're interested in installing and testing the latest Apache Polaris open-source release, Dremio has published a hands-on tutorial that guides you through the installation process and integration with Apache Spark and Apache Iceberg. —> Read More

⚙️ Technical Deep Dive

👉 Parquet vs ORC: Choosing the Right Format for Data Lakehouse

In a recent Apache Hudi blog post, the author compares Parquet and ORC, two of the most popular serialisation frameworks for data lakes and open table formats. The post argues that Parquet delivers better performance for read-heavy, complex analytical use cases where query performance is crucial, while ORC offers a more balanced approach for both read and write performance with superior compression, making it a better fit for general-purpose data storage in Hudi. —> Read More

👉 Overview of Amazon MSK Tiered Storage

Amazon recently published a post, explaining how the new Kafka tiered storage in Amazon MSK (Managed Kafka Service) enhances scalability and resiliency. With the new decoupled storage and compute architecture, the system benefits from faster broker recovery, improved load balancing, and virtually unlimited scalability. —> Read More

💬 Community Discussions

There have been several widely discussed threads including link and link on Reddit last month about data engineering roles and the challenges of applying for Data Engineering jobs. Based on comments from both candidates and hiring managers (including CTOs), the situation seems two-fold.

On one hand, the market is flooded with unqualified candidates—often with minimal skills from low-quality bootcamps and online courses—making it hard for companies to find qualified engineers. On the other hand, there are vague and unclear job descriptions, leaving data engineers unsure of what is expected of them once hired. This has created frustration on both sides.

🔎 Case Studies

👉 Evolution of Apache Flink Architecture at Airbnb

Airbnb published a post detailing the evolution of their Apache Flink architecture. Initially, they deployed Flink jobs on Hadoop YARN with Airflow as the scheduler in 2018. Today, they’ve moved to deploying Flink jobs on Kubernetes, eliminating the need for a job scheduler. —> Read More

👉 Pinterest's Adoption of StarRocks for Real-Time Analytics

Pinterest shared their recent adoption of StarRocks, a real-time OLAP engine, for their real-time analytics platform. They chose StarRocks for its features like support for standard SQL, joins, sub-queries, and materialized views—capabilities not readily available in other real-time OLAP engines like Druid. Back in 2021, Pinterest published details about managing a large Druid fleet with 2,000 nodes in a multi-cluster setup. —> Read More

👉 The Rise of New Real-time OLAP Engines

On that note, while Apache Druid, Pinot, and ClickHouse have dominated the open-source real-time OLAP space in recent years, we’re now seeing increased adoption of newer engines like Apache Doris and StarRocks, the latter being a fork of Doris. For a detailed comparison between StarRocks and Doris, check out this blog post from StarRocks Engineering.

👉 Uber’s Hadoop-to-Cloud Migration

Uber, which operates one of the largest on-premise Hadoop clusters, has recently begun migrating to the cloud, starting with a key architectural shift—replacing the HDFS file system with Google Cloud Storage, while still running the rest of their stack on IaaS. One of the challenges in migrating from Hadoop to the cloud is transitioning Hadoop’s security features, such as delegation tokens and Kerberos authentication, to Google Cloud’s token-based security. Uber discusses how they tackled these security migration challenges in this article. —> Read More

📣 Vendors News & Announcements

👉 Fivetran Integration with Snowflake’s Polaris Catalog

Just days after Snowflake open-sourced the Polaris catalog service on GitHub, Fivetran, a leading SaaS provider for data integration, announced its upcoming integration with the newly open-sourced Polaris data catalog. This integration aims to develop a managed catalog solution for Fivetran’s Managed Data Lake Service. —> Read More

👉 Databricks LakeFlow Connect for Automated Data Ingestion

Databricks announced the public preview of LakeFlow Connect, an automated incremental data ingestion service for sources like SQL Server and Salesforce. Built on Delta Live Tables, LakeFlow Connect enables incremental data ingestion using CDC (Change Data Capture). This marks another step by major vendors toward automating data engineering tasks. —> Read More

👉 Databricks Lakehouse Federation Across AWS, Azure, and GCP

Databricks also announced the general availability of Lakehouse Federation in Unity Catalog across AWS, Azure, and GCP last month. This mirrors the strategy of other top cloud vendors to offer a unified analytical platform with centralised data discovery and governance, providing a unified view of enterprise data across multiple storage engines and cloud platforms. —> Read More

👉 ClickHouse Acquisition of PeerDB for Real-Time Postgres Ingestion

ClickHouse, Inc. announced the acquisition of PeerDB, a provider of Change Data Capture (CDC) for Postgres databases. This move aims to integrate and streamline real-time data ingestion from transactional databases like Postgres into the ClickHouse OLAP engine. —> Read More

🎥 Conferences & Events

👉 Kafka Current 2024 Key Notes

The two-day Kafka Current 2024 event, organised by Confluent, took place last week in Austin. Keynotes from both Day 1 and Day 2 have already been published on YouTube: Keynote Day 1 | Keynote Day 2

👉 Open Source Data Summit Virtual Conference

The Open Source Data Summit Virtual Conference will be held on October 2nd. If you're interested, you can register for free at opensourcedatasummit.com. The event will feature numerous discussions on data lakehouses and the role of open table formats in modern data architectures.

✨ Featured - Netflix Maestro Workflow Engine

The crowded field of open-source engines has just welcomed a new player: Maestro, recently open-sourced by Netflix!

Netflix asserts that Maestro is a highly scalable and flexible scheduler capable of managing large-scale heterogeneous workflows, including ML training and data pipelines. It supports flexible execution logic, such as Docker images and notebooks, and accommodates various workflow patterns, including cyclic and acyclic (DAGs).

One of its standout features is the foreach pattern, which is particularly useful for repetitive tasks like ML model training and data backfilling—something that would typically require separate job runs on a scheduler like Airflow to backfill daily ingested source data. Maestro also offers multiple domain-specific languages (DSLs) for defining workflows declaratively using YAML files, a feature that would need to be custom-built on top of Airflow.

Since its open-source release in July, the project has already garnered 3,000 stars on Github. Netflix had previously covered the internals and use cases implemented with this engine, and their latest blog post provides a comprehensive overview of Maestro's features and supported workflow patterns.

Additionally, this blog post provides a thorough comparison between Airflow and Maestro, complete with practical examples and code snippets.

💡 Trends & Insight

👉 The State of Modern Data Stack in 2024

In 2024, there has been much discussion about the decline of the Modern Data Stack (MDS). Concerns have been raised about the economics of the Modern Data Stack, and the term itself is being recycled, much like previously hyped concepts such as "Big Data". Some experts believe that many MDS startups are doomed to extinction. As Matt Turck pointed out, the Modern Data Stack was largely a marketing concept and an alliance among several startups across the data value chain. In a recent blog post, Ananth explores the history, decline, and the emerging post-MDS era. —> Read more

👉 Embracing 'Bring Your Own Compute' with DuckDB

An interesting discussion took place between MotherDuck’s co-founder and CEO and Fivetran’s co-founder and CEO about the future of big data and single-node or laptop-sized analytics using DuckDB. With advancements in hardware, we might witness a new shift towards local execution, where computing is fully or partially pushed to the user’s machine, introducing the concept of Bring Your Own Compute for analytics. —> Watch the interview

📡 Open Source News

👉 Apache Kafka 3.8 Release

Apache Kafka 3.8 has been Released. Confluent blog provided a summary of the the new features and improvements in this release. —> Read more

👉 Delta Lake 4.0 New Features

Delta Lake 4.0 Preview was announced in June. A blog post highlights the new important features such as Change Data Feed (CDF), Liquid Clustering and new monitoring features of this release with some practical examples. —> Read More

👉 DAG Factory Project Takeover by Astronomer

Astronomer has announced taking over the open-source project DAG Factory, a Python library for authoring Airflow DAGs declaratively using YAML configuration files. Providing a thin no-code abstraction layer on top of Airflow has become a common practice among tech companies to reduce the engineering effort required to create DAGs, standardise pipeline creation for common use cases like data transformation, and make the process more self-service. —> Read more

🛠 Practical Data Engineering

👉 Consistent Data Modeling and Naming Conventions

Implementing a consistent data modeling framework, such as standardised naming conventions, is crucial to maintaining a healthy data platform and ensuring long-term scalability, even when there is turnover among engineers. Mike discusses some of the key aspects and best practices for data warehouse modeling, including effective naming conventions for tables and schemas. —> Read more

👉 State of CI/CD for Data Pipelines

LakeFS published a comprehensive overview of implementing Continuous Integration/Continuous Delivery (CI/CD) for data pipelines, focusing on the Write-Audit-Publish (WAP) ingestion pattern. The article explores various options and tools available in the market, offering insights into how to effectively integrate CI/CD practices into data workflows. --> Read more

👉 Data Reconciliation Techniques and Best Practices

Datafold has published a three-part series on data reconciliation, a crucial subset of data quality. The series covers use cases, techniques, challenges, and best practices for performing data reconciliation across data sources and targets, with the goal of ensuring data accuracy and completeness. —> Read more

👉 dbt Beyond the Marketing Hype

There have been many blog posts and discussions about the hype dbt has generated over the past few years. The author of this blog post takes a different approach, discussing the challenges of performing data transformation in data warehouses and how dbt can address them. It starts with real problems and then explores how tooling, specifically dbt, provides solutions—rather than starting with the tool (because it's popular and everyone is talking about it) and then searching for problems it can solve. The post also offers a clear definition of what dbt actually does:

dbt works by abstracting common data-warehouse patterns into config-driven automation and providing a suite of tools to simplify SQL transformations, tests, and documentation.

⚙️ Technical Deep Dive

👉 Evolution of Debezium's Internal Engine

While in many streaming CDC data architectures, Debezium plugins are primarily used within the Kafka Connect framework and runtime, it is also possible to run Debezium connectors outside the Kafka ecosystem. This can be done by embedding the Debezium engine in internal applications or by using the standalone Debezium Server, which is now a separate project on GitHub. This Debezium blog discusses the evolution of Debezium's internal engine, starting with the initial EmbeddedEngine implementation, which was mainly built for testing, and the new AsyncEmbeddedEngine, which addresses the shortcomings of the previous implementation. —> Read more

👉 A guide on Concurrency Levels of Apache Airflow

One of the most confusing aspects of the Apache Airflow engine, especially for newcomers, is how concurrency is applied at different levels, such as the Airflow scheduler, DAG level, and task level, and how their combination can impact overall workflow performance. The configuration parameters in the config file of early released versions added to this confusion, with names that often seemed unrelated. In a recent blog post, Google provides a comprehensive overview of the various concurrency levels in Airflow, with a particular focus on its managed Airflow service. —> Read more

👉 Apache Airflow Software Architecture

A helpful guide posted by Apache Airflow's blog with visuals that illustrates the key underlying components of the Apache Airflow software architecture and how they interact within the system. —> Read more

Speaking of Airflow, do you...!?

👉 Bringing GenAI and LLMs to Flink Streaming Pipelines

Confluent blog provided an overview of a new Flink AI feature, which allows streaming data pipelines to invoke AI models, including generative AI (GenAI) large language model (LLM) endpoints (such as OpenAI and Google Vertex AI), directly from Flink SQL statements. This enables tasks like AI model inference, regression, and classification to be seamlessly integrated into real-time data processing workflows. —> Read more

🔎 Case Studies

👉 Pinterest's Migration from HBase to TiDB

Pinterest shared their journey of replacing HBase storage with a modern, scalable open-source database system that meets their requirements for reliability, performance, tunable consistency, and robust CDC support. They ultimately chose TiDB as the solution. We are seeing more stories of companies exploring alternatives to HBase due to its limitations and maintenance overhead. —> Read more

👉 Slack’s Migration to EMR 6

Slack discusses their migration from EMR 5 with Spark 2 to EMR 6 with Hive 3 and Spark 3 on AWS, highlighting the performance and reliability improvements achieved in their data pipelines, which are developed using Apache Spark and scheduled on Airflow. —> Read more

💬 Community Discussions

There was a recent discussion on Reddit on how fast data engineering is progressing. The consensus among most commenters is that while tools, storage systems, and processing frameworks may evolve rapidly, the fundamentals of data engineering remain consistent. These fundamentals include data integration, data modeling, and the processes of extracting, transforming, and loading data (ETL).

For aspiring data engineers, the key takeaway is to invest time and effort in mastering these basics and fundamentals rather than focusing solely on becoming an expert in specific tools. While vendors are striving to automate the data engineering lifecycle as much as possible (as seen with the latest Databricks offerings), a strong understanding of the fundamentals will always be valuable and help you stand out in the field.

📣 Vendors News & Announcements

👉 Snowflake's New Cortex Search Feature

Snowflake announced a new feature called Cortex Search (currently in Public Preview) in July 2024. This search service is designed for unstructured data, such as text, and enables enterprises to deploy Retrieval-Augmented Generation (RAG) applications using Snowflake, allowing them to customise generative AI applications with proprietary data. —> Read more

👉 Databricks Mosaic AI Model Training

Around the same time, Databricks announced the support for Mosaic AI Model Training, which streamlines the fine-tuning of general-purpose open-source LLM and GenAI models, such as Llama 3 and Mistral, using enterprise data. Databricks recommends a new approach for training LLM models with enterprise data called Retrieval Augmented Fine-tuning (RAFT), which combines both Retrieval-Augmented Generation (RAG) and model fine-tuning. —> Read more

👉 Release of Confluent Platform 7.7

Confluent announced the release of Confluent Platform 7.7, built on Apache Kafka 3.7. This update introduces significant features, including Confluent Platform for Apache Flink, a fully managed and serverless stream processing service (currently in Limited Availability), as well as a self-managed HTTP Source connector for ingesting data from external APIs. —> Read more

Introduction

Welcome to the Data Engineering Digest (DLD) newsletter series. This will be a periodic roundup of the latest and greatest in the world of data & data engineering in particular. We'll deliver a curated selection of finest news articles, blog posts, tutorials, discussions, and more within the data landscape.

✨ Featured

In a recent blog post I explored the difference between generic data lakehouses offered by some vendors, and open data lakehouses built on the foundation of open source tools and technologies. The Apache Hudi blog published a post providing an overview of the open data lakehouse architecture. It details the architecture layers, components, key technologies, advantages over previous data lake architectures, and use cases for implementing an open data lakehouse. —> Read more

📡 Open Source News

👉 Debezium Latest Official Release

Debezium 2.7.0.Final has been released. Some outstanding 140 issues have been fixed along with many new features and improvements in the core component as well as the stand-alone connectors. —> Read more

👉 Submission of OneTable to Apache Foundation

OneHouse news on submission of XTable (formerly known as OneTable) to the Apache Software Foundation Incubator was a major recent announcement. Top cloud vendors such as Microsoft and Google are already integrating XTable into their analytics platforms, Microsoft Fabric and BigLake respectively, to provide a unified logical lakehouse with interoperability between different open table formats. —> Read more

🛠 Practical Data Engineering

👉 Spark Repartition() Function

Spark pipelines often employ the df.repartition() function to optimise data processing, especially by consolidating small partitions before loading data into target storage. It's essential to remember that repartitioning in Spark is a unit of parallelism and data distribution used by the distributed compute engine, not necessarily a full bucketing or SQL-style group by operation. A blog post explains how Spark repartitioning works and what it actually does. --> Read more

👉 Apache XTable + Airflow

AWS published a blog post demonstrating how to use Apache XTable to convert open table format metadata to other formats. The blog post features a custom Airflow operator, XtableOperator(), designed for batch pipeline translations on the AWS platform. The operator's code is available on Github. This development suggests that unified open table format adoption is gaining some momentum. —> Read more

⚙️ Technical Deep Dive

👉 Apache Paimon’s Internal Design

While the 'big three' open table formats – Apache Hudi, Iceberg, and Delta Lake – dominate the market and discussions, Apache Paimon, a more recent “Flink table format”, has received less attention. If you're interested in learning more about Apache Paimon, a comprehensive blog post by Giannis delves into its design goals, internals, and key features. —> Read more

👉 How dlt Works Under the Hood

Discussions have been ongoing regarding the potential use of new open source data ingestion tools like Data Load Tool (dlt) as a replacement for more established ones like Airbyte in certain use cases (ex API data integration). A dlt project contributor has published a blog post detailing the internal data pipeline design and core functions of dlt, including data extraction, normalisation, and loading. The latest version leverages Apache Arrow's efficient in-memory data structure to optimise the entire pipeline. —> Read more

👉 Yet Another Kafka Explanation

There's a wealth of resources available explaining Kafka's architecture and internals. I found a recent blog post series on the topic, which provides clear and concise explanations of the concepts, accompanied by helpful visuals for those unfamiliar with Kafka's design and operation.

Kafka Architecture Overview | Design elements | Kafka Producer | Kafka Consumer

👉 DuckDB's Internal Memory and Buffer Management

If you've used DuckDB or are exploring its capabilities, you might wonder how it handles large datasets without memory limitations, a common issue with some Python dataframes like Pandas. DuckDB’s official blog has recently covered the engine's internal memory and buffer management. It explains how DuckDB leverages streaming execution to process queries without fully loading CSV or Parquet files into memory, and utilises disk spilling when intermediate results exceed memory capacity. —> Read more

👉 Snowflake's Micro Partitioning Internal Design

If you've been using Snowflake at your company, you're likely familiar with its internal partitioning feature called micro-partitioning. This process automatically divides tables into micro-partitions of 50 MB and 500 MB, organising data in a columnar format within each micro-partition. A concise and excellent blog post provides a clear explanation of micro-partitioning's internal design, complete with helpful visuals. —> Read more

👉 Overview of Kafka's Tiered Storage Design

Kafka 3.6, released in 2023, introduced a highly anticipated feature: Tiered Storage. This feature currently supports local and remote storage tiers, enabling the movement of inactive segments to a configurable deep storage solution like HDFS or S3 based on local retention settings. This provides a cost-effective and scalable way to retain historical data. Uber is credited with driving the tiered storage proposal [KIP-405], discussing the internals of the tiered storage architecture. —> Read more

🔎 Case Studies

👉 Implementation of dlt in Production

It's inspiring to hear about individuals and teams embracing new open-source tools and technologies in production environments. Data Load Tool (dlt) is a relatively new ETL library covered earlier, that has been adopted by some for production workloads. Alexander from Dataops explores the advantages and disadvantages of using dlt compared to more established data integration tools like Airbyte. —> Read more

👉 Notion's Data Architecture Evolution

Notion has unveiled their new data lakehouse architecture. They chose Apache Hudi as their table format due to its efficient incremental data ingestion capabilities, making it suitable for their update-heavy workloads. The architecture also incorporates event-based CDC ingestion using Debezium and Kafka. —> Read more

💬 Community Discussions

👉 This career advice from a senior data engineer highlights a key point in one of Reddit's highest-rated data engineering discussions in July:

While flashy tools and platforms come and go, a strong foundation in low-level skills like Bash, Git, SQL, pure Python development, and containerisation will take you much further. Juniors who prioritise these foundational skills before diving into advanced tools and stacks will be better-positioned for success.

🎥 Conferences & Events

👉 The annual virtual PrestoCon 2024 day organised by Linux Foundation/Presto Foundation took place in June, discussing topics like Presto 2.0 native C++ engine, and Presto usage at companies like Uber. A recap of the event and the main sessions is provided in this article. All the recorded 24 sessions can be found on Youtube.