Code Monkey home page Code Monkey logo

sparkdatalog's Introduction

SparkDatalog

A Datalog API for Spark, which allows for mix Datalog queries into Spark computations. Written entirely in Scala.

Example

// Assuming we have the following RDDs (for example read from HDFS):
val edgesRdd = sc.parallelize(Seq((1, 2, 1), (1, 3, 5), (2, 3, 1)))
val sourceRdd = sc.parallelize(Seq(0))

// Compute shortests paths from the source node using Spark Datalog API:

//   1. Create a Database from Relations built from RDDs.    
val database = Database(
  Relation.ternary("Edge", edgesRdd),
  Relation.unary("IsSource", sourceRdd))

//   2. Execute a Datalog query on the database, producing a new Database.
val query = """
    |declare Path(int v, int dist aggregate Min).
    |Path(x, d) :- IsSource(s), Edge(s, x, d).
    |Path(x, d) :- Path(y, da), Edge(y, x, db), d = da + db.
  """.stripMargin
val resultDatabase: Database = database.datalog(query)

//   3. Retrieve the result from the new Database.
val resultPathsRdd: RDD[Seq[Int]] = resultDatabase("Path")


// We can now save the paths RDD to distributed storage
// or perform further computations on it.

// We can of course also print it to stdout:
print(resultPathsRdd.collect().map("Path(" + _.mkString(", ") + ")").mkString("\n"))
// Outputs:
//   Path(3, 2)
//   Path(2, 1)

Quick start

Check out SparkDatalog source with the example project:

git clone [email protected]:marekrogala/sparkdatalog.git

Run the example program:

cd sparkdatalog/sparkdatalog
sbt run

To compile and run the program, you need to have SBT installed (http://www.scala-sbt.org/download.html).

sparkdatalog's People

Contributors

marekrogala avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

sparkdatalog's Issues

Feasibility of Datalog on top of Spark3.0

Hello,
I wanna ask if this is related to the Bigdatalog proposed by "Alexander Shkapsk".

I want to ask about the feasibility to build this module on top of Spark 3.0 .

Thanks in advance for your answer!
Best,
Ragab

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.