oap-project Goto Github PK

arrow

Apache Arrow is a cross-language development platform for in-memory data. It specifies a standardized language-independent columnar memory format for flat and hierarchical data, organized for efficient analytic operations on modern hardware. It also provides computational libraries and zero-copy streaming messaging and interprocess communication. Languages currently supported include C, C++, Java, JavaScript, Python, and Ruby.

arrow-data-source

Spark DataSouce plugin for reading files from various formats like Parquet into Arrow compatible columnar vectors.

cloudtik

Cloud Scale Platform for Distributed Analytics and AI

gazelle_plugin

Native SQL Engine plugin for Spark SQL with vectorized SIMD optimizations.

gluten-trino

Gluten: Plugin to Boost Trino's Performance

libhdfs3

HDFS file read access for ClickHouse

libhdfs3-downstream

a native c/c++ hdfs client (downstream fork from apache-hawq)

oap-mllib

Optimized Spark package to accelerate machine learning algorithms in Apache Spark MLlib.

oap-project.github.io

The OAP project web site

oap-tools

Tools for building, packaging, and OAP public cloud integrations such as AWS EMR, Google Dataproc and K8S.

pmem-common

Common library for accessing PMEM native library functions including memkind, vmemcache and so on.

pmem-shuffle

Spark* Shuffle plugin for support shuffling through remote persistent memory over fabrics, which leverages the RDMA network and remote persistent memory (for read) to provide extremely high performance and low latency shuffle solutions for Spark*.

pmem-spill

Spark plug-in package for accelerating Spark runtime spill functions using PMem such as RDD cache PMem extension.

protobuf

A Intel customized Protocol Buffers - Google's data interchange format

raydp

RayDP provides simple APIs for running Spark on Ray and integrating Spark with AI libraries.

recdp

remote-shuffle

Spark* shuffle plugin for support shuffling data through a remote Hadoop-compatible file system, as opposed to vanilla Spark's local-disks.

solution-navigator

Example solutions or code for using OAP features.

spark-ai-kit

Gluten: Plugin to Double SparkSQL's Performance

sql-ds-cache

Spark* plug-in for accelerating Spark* SQL performance by using cache and index at SQL data source layer.

velox

A new C++ vectorized database acceleration library aimed to optimizing query engines and data processing systems.