🚀 Apache Top-Level Project

Supercharge SparkSQL with Native Execution

Apache Gluten is a middle layer that offloads JVM-based SQL engines to native engines like Velox and ClickHouse — delivering up to 3x+ performance with zero query changes.

Get Started View on GitHub Downloads

3.3x

TPC-H Speedup

3.0x

TPC-DS Speedup

23x

Max Single-Query Speedup

15+

Production Adopters

Why Gluten?

Gluten bridges the gap between Spark's scalability and native engine performance — no query changes required.

⚡

Native Execution

Offloads compute-intensive processing from the JVM to highly optimized native C++ engines for dramatic speedups.

🔌

Drop-in Plugin

Works as a Spark plugin — no changes to your DataFrames, SQL queries, or application code. Just add a JAR.

🔀

Multi-Backend Support

Switch between Velox and ClickHouse backends based on your workload. Extensible to support more native engines.

📊

Columnar Processing

End-to-end columnar data pipeline with columnar shuffle, avoiding expensive row-columnar conversions.

🛡️

Automatic Fallback

Gracefully falls back to vanilla Spark for unsupported operators, ensuring correctness on every query.

📈

Full Observability

Native engine metrics are surfaced in the Spark UI, giving you visibility into performance and bottlenecks.

Architecture

Spark physical plans are converted to Substrait and executed by native backends, with results returned via the Spark Columnar API.

Quick Start

Grab a bundle JAR from the downloads page and enable Gluten with a single Spark configuration change.

spark-shell \
  --master yarn --deploy-mode client \
  --conf spark.plugins=org.apache.gluten.GlutenPlugin \
  --conf spark.memory.offHeap.enabled=true \
  --conf spark.memory.offHeap.size=20g \
  --conf spark.shuffle.manager=org.apache.spark.shuffle.sort.ColumnarShuffleManager \
  --jars gluten-velox-bundle-spark3.5_2.12-centos_7_x86_64-1.6.0.jar

Read the full documentation →