The Blaze accelerator for Apache Spark leverages native vectorized execution to accelerate query processing. It combines the power of the Apache Arrow-DataFusion library and the scale of the Spark distributed computing framework.

Blaze takes a fully optimized physical plan from Spark, mapping it into DataFusion's execution plan, and performs native plan computation in Spark executors.

Blaze is composed of the following high-level components:

• Blaze Spark Extension: hooks the whole accelerator into Spark execution lifetime.
• Native Operators: defines how each SparkPlan maps to its native execution counterparts.
• JNI Gateways: passing data and control through JNI boundaries.
• Plan SerDe: serialization and deserialization of DataFusion plan with protobuf.
• Columnarized Shuffle: shuffle data file organized in Arrow-IPC format.

Based on the inherent well-defined extensibility of DataFusion, Blaze can be easily extended to support:

• Various object stores.
• Operators.
• Simple and Aggregate functions.
• File formats.

We encourage you to extend DataFusion capability directly and add the supports in Blaze with simple modifications in plan-serde and extension translation.

To build Blaze, please follow the steps below:

1. Install Rust

The underlying native execution lib, DataFusion, is written in Rust Lang. So you're required to install Rust first for compilation. We recommend you to use rustup.

curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh

2. Check out the source code.

git clone [email protected]:blaze-init/blaze.git
cd blaze

3. Build the project.

You could either build Blaze in debug mode for testing purposes or in release mode to unlock the full potential of Blaze.

./gradlew -Pmode=[dev|release-lto] build

After the build is finished, a fat Jar package that contains all the dependencies will be generated in the target directory.

This section describes how to submit and configure a Spark Job with Blaze support.

You could enable Blaze accelerator through:

$SPARK_HOME/bin/spark-[sql|submit] \
  --jars "/path/to/blaze-engine-1.0-SNAPSHOT.jar" \
  --conf spark.sql.extensions=org.apache.spark.sql.blaze.BlazeSparkSessionExtension \
  --conf spark.executor.extraClassPath="./blaze-engine-1.0-SNAPSHOT.jar" \
  .... # your original arguments goes here

At the same time, there are a series of configurations that you can use to control Blaze with more granularity.

We periodically benchmark Blaze locally with a 1 TB TPC-DS Dataset to show our latest results and prevent unnoticed performance regressions. Check Benchmark Results with the latest date for the performance comparison with vanilla Spark.

Currently, you can expect up to a 2x performance boost, cutting resource consumption to 1/5 within several keystrokes. Stay tuned and join us for more upcoming thrilling numbers.

We also encourage you to benchmark Blaze locally and share the results with us. 🤗

Currently, there are still several operators that we cannot execute natively:

• Aggregate. Relies on apache/datafusion#1570.
• Join with an optional filter condition. Relies on apache/datafusion#2509.
• Broadcast HashJoin.
• Window.

We use segmented Arrow-IPC files to express shuffle data. If we could apply IPC compression, we would benefit more from Shuffle since columnar data would have a better compression ratio. Tracked in #4.

We would like to have a high-performance JVM-UDF invocation framework that could utilize a great variety of the existing UDFs written in Spark/Hive language. They are not supported natively in Blaze at the moment.

We're using Discussions to connect with other members of our community. We hope that you:

• Ask questions you're wondering about.
• Share ideas.
• Engage with other community members.
• Welcome others and are open-minded. Remember that this is a community we build together 💪 .

Blaze is licensed under the Apache 2.0 License. A copy of the license can be found here.