DataFrames are useful constructs for data scientists, popularized by R and pandas.
Implementations of these packages are not natively distributed - their DataFrames struggle with big data. bigdf is a dataframe on top of Apache Spark. It is written as an internal Scala DSL with the look and feel of Pandas to make it easy for those familiar with Pandas to use it.
A gentle(but outdated) introduction is available at:
- Choose a repos directory (make one if it doesn't exist)
export $REPOS = ~/Documents/repos- Install scala and sbt (brew instructions below)
$ brew install scala
$ brew install sbt- clone the bigdf repo
$ cd $REPOS
$ git clone [email protected]:ayasdi/bigdf.git- sbt update, package, and test(optional) in the bigdf directory.
$ cd bigdf
$ sbt update
$ sbt test
$ sbt assembly- Start a shell and begin playing. You need spark 1.4.x
$ cd $REPOS
$ ./spark/bin/spark-shell --jars ./bigdf/target/scala-2.10/bigdf-assembly-*.jar - https://github.com/AyasdiOpenSource/bigdf/wiki/ for examples
- Look at DFTest.scala for usage
$cd $REPOS
$git clone [email protected]:mattpap/IScala.git
$cd IScala
$sbt
>++ 2.10.4
>assembly
>^D$ipython profile create scalaAdd the following line to the three files ~/.ipython/profile_scala/ipython*_config.py
c.KernelManager.kernel_cmd = ["java", "-jar", "$ISCALA_PATH/lib/IScala.jar", "--connection-file", "{connection_file}", "--parent"]"To start notebook
$cd <my notebooks directory>
$ipython notebook --profile scalaTo install spark and bigdf to your local mvn repos
$brew install maven
$mvn install:install-file -Dfile=$REPOS/bigdf-transforms/lib/spark-assembly-1.2.0-SNAPSHOT-hadoop1.0.4.jar -DgroupId=org.apache -DartifactId=spark-assembly_2.10 -Dversion=1.2.0-SNAPSHOT-hadoop1.0.4 -Dpackaging=jar
$ mvn install:install-file -Dfile=$REPOS/bigdf-transforms/bigdf/target/scala-2.10/bigdf-assembly-0.2.jar -DgroupId=com.ayasdi -DartifactId=bigdf-assembly_2.10 -Dversion=0.2 -Dpackaging=jarTo start using bigdf in a notebook
%libraryDependencies += "org.apache" %% "spark-assembly" % "1.2.0-SNAPSHOT-hadoop1.0.4"
%libraryDependencies += "com.ayasdi" %% "bigdf-assembly" % "0.2"
%resolvers += "Local Maven Repository" at "file:///Users/mohit/.m2/repository"
%updateimport org.apache.spark.SparkContext
import org.apache.spark.SparkContext._
import com.ayasdi.bigdf._
val sc = new SparkContext("local[*]", "bigdf-intro")
val df = DF(sc, "/Users/mohit/Downloads/faredata2013/trip_fare_1.csv", ',', true)
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent