jiangyu Goto Github PK

adfs-1

ADFS (Ali Distributed File System) is an evolutional version of Hadoop which delivers high availability, auick-restart and other features.

avro2tf

Avro2TF is designed to fill the gap of making users' training data ready to be consumed by deep learning training frameworks like TensorFlow.

bash

some bash code to use

distcpv2-0.20.203

Rewrite of Hadoop DistCp, for Hadoop 0.20.203+. Support for multiple copy-strategies. Code also reads better.

docker-java

Java Docker API Client

fb-raid-refactoring

Refactoring of the RAID code base to make new Erasure Codes easily pluggable

flume

Mirror of Apache Flume

hadoop

Mirror of Apache Hadoop

hadoop-lzo

Patched, refactored version of code.google.com/hadoop-gpl-compression for hadoop 0.20

hadooplogdetect

Detect hadoop log from aggregation logs using spark

haoel.github.io

imagesummary

Get the summary information from hdfs image without parse every inode to avoid using too much memory

kubernetes

Production-Grade Container Scheduling and Management

magix

A Web Application Framework

presto

The official home of the Presto distributed SQL query engine for big data

presto-1

Home of the community managed version of Presto, the distributed SQL query engine for big data, under the auspices of the Presto Software Foundation.

s3fs

S3 Filesystem

scalalearn

scaling-hdfs-namenode

This work aims to re-engineer the Hadoop Distributed File System (HDFS) so that it can be 1) highly available, and 2) horizontally scalable. This is achieved by replacing the central master server with a distributed real-time database (in our implementation, MySQL Cluster).

scribe

Scribe is a server for aggregating log data streamed in real time from a large number of servers. It is designed to be scalable, extensible without client-side modification, and robust to failure of the network or any specific machine.

slidebtn

spark

Apache Spark

spark-deep-learning

Deep Learning Pipelines for Apache Spark

sparklda

implement LDA algorithm using spark

tensorflow

An Open Source Machine Learning Framework for Everyone