See description at top of GitHub page.

http://distributedlog.incubator.apache.org

Hi,

here I propose that you add Apache YARN into the list. In my point of view it's more-on-less a standalone component, but a very fundamental one. Understanding what it does could be crucial in understanding the ecosystem.

And there are a lot's of references to it in the descriptions :)

First of all, this list is an amazing compilation !

Perhaps Ganglia ( http://ganglia.sourceforge.net/ ) should also be included since it is a pretty standard distributed monitor tool in Hadoop clusters.

And now Apache Flink is an Apache incubator project.

Cloudbreak is a cloud agnostic Hadoop as a Service API. Abstracts the provisioning and ease management and monitoring of on-demand clusters.

http://sequenceiq.com/cloudbreak/

This might be considered to be "above" the level of Hadoop... but I just thought I'd ask since Hortonworks is involved with Cloudbreak now...

These two projects are quite active right now.

Crail

Apache Crail is a fast multi-tiered distributed storage system designed from ground up for high-performance network and storage hardware.
It marks the backbone of the Crail I/O architecture, which is described in more detail on crail.incubator.apache.org.
The unique features of Crail include:

• Zero-copy network access from userspace
• Integration of multiple storage tiers such DRAM, flash and disaggregated shared storage
• Ultra-low latencies for both meta data and data operations. For instance: opening, reading and closing a small file residing in the distributed DRAM tier less than 10 microseconds, which is in the same ballpark as some of the fastest RDMA-based key/value stores
• High-performance sequential read/write operations: For instance: read operations on large files residing in the distributed DRAM tier are typically limited only by the performance of the network
• Very low CPU consumption: a single core sharing both application and file system client can drive sequential read/write operations at the speed of up to 100Gbps and more
• Asynchronous API leveraging the asynchronous nature of RDMA-based networking hardware
• Extensible plugin architecture: new storage tiers tailored to specific hardware can be added easily

Arrow

Apache Arrow is a cross-language development platform for in-memory data. It specifies a standardized language-independent columnar memory format for flat and hierarchical data, organized for efficient analytic operations on modern hardware. It also provides computational libraries and zero-copy streaming messaging and interprocess communication. Languages currently supported include C, C++, Java, JavaScript, Python, and Ruby.

Hi Jroman

Do you have plan to share the updated version of the table?

• Riak and RiakCS as distributed KV and object/file store
• FoundationDB as multiparadigm distributed DB
• OrientDB is a primarily graph, multiparadigm DB

While the cephfs hadoop jars linked of the ceph website do indeed indicate a need for Hadoop 1.1.x, you can build these jars yourself, and it will build against a Hadoop 2.x distro, see: https://github.com/ceph/cephfs-hadoop.

Apache Heron is a distributed programming/processing system. It should be also added under the Distributed Programming section.
Link: https://github.com/apache/incubator-heron

Was reading through the bit about Hadoop + LustreFS and noticed this sentence:

... Intel's
noticed this and, in version 2.5 of its Hadoop distribution that it quietly
released last week...

Also, "Intel's" should probably be "Intel" or "Intel's Engineers" (?)