Knowledge Sharing (KS) - Benchmark of data processing engines

Table of Content (ToC)

• Knowledge Sharing (KS) - Benchmark of data processing engines
  • Table of Content (ToC)
  • Overview
  • References
  • Articles
    • Light ETL Python engines
    • Accelerating Apache Spark's Execution Engine
    • Accelerating Apache Spark with Gluten and Velox
    • Polars vs DuckDB

Created by gh-md-toc

Overview

This project aims at benchmarking a few data processing engines (e.g., DuckDB, Spark, Polars, Daft).

Inspiration: Todo MVC

Even though the members of the GitHub organization may be employed by some companies, they speak on their personal behalf and do not represent these companies.

References

• Todo MVC home page
• Material for the Data platform - Data access to Databricks data lakehouse
• Data Engineering Helpers - Knowledge Sharing - Cheat sheets
  • Data Engineering Helpers - Knowledge Sharing - Python
  • Data Engineering Helpers - Knowledge Sharing - Spark
  • Data Engineering Helpers - Knowledge Sharing - DuckDB
  • Data Engineering Helpers - Knowledge Sharing - SQLMesh
  • Data Engineering Helpers - Knowledge Sharing - dbt
• Material for the Data platform - Architecture principles
• Material for the Data platform - Modern Data Stack (MDS) in a box
• Material for the Data platform - Data life cycle
• Material for the Data platform - Data contracts
• Material for the Data platform - Metadata
• Material for the Data platform - Data quality

Articles

Light ETL Python engines

• Author: Mimoune Djouallah (Mimoune Djouallah on LinkedIn, Mimoune Djouallah on GitHub)
• Date: Jan. 2026
• Git repository with a Jupyter notebook and fully reproducible scripts
  • Git repository with the data sets
• Post on LinkedIn

Accelerating Apache Spark's Execution Engine

• Title: Accelerating Apache Spark's Execution Engine
• Author: Dipankar Mazumdar
• Date: Dec. 2025
• Post on LinkedIn

Accelerating Apache Spark with Gluten and Velox

• Title: Accelerating Apache Spark with Gluten & Velox
• Author: Angel Conde (Angel Conde on LinkedIn, Angel Conde on Medium)
• Date: Sep. 2025
• Link to the article on Medium
• Companion Git repository. It features a benchmark with:
  • Public, generated, datasets containing a fact table and dimension tables
  • Several queries representing typical analytics workload:
    • Query A — Heavy multi‑aggregation: Groups the fact table by country_id and channel_id and computes counts, sums, averages, standard deviation and approximate percentiles. This pattern stresses hash aggregation, projection and filter operators.
    • Query B — Rollup (cube) aggregation: Joins the fact table with a date dimension and uses rollup(date_key, country_id, product_id) to compute revenue, quantity and average discount across multiple grouping levels.
    • Query C — Star schema join and top‑K sort: Joins the fact table with broadcast dimensions (countries and channels) and the date dimension, computes gross and net revenue, and orders by gross descending, taking the top 5,000 rows. Broadcast joins and top‑K sorts test Velox’s vectorized join and sort operators.

Polars vs DuckDB

• Title: CSV, GZip, S3, Python (Polars vs DuckDB)
• Date: Nov. 2025
• Author: Daniel Beach (Daniel Beach on LinkedIn, Daniel Beach on Substack)
• Link to the article on Substack