Code Monkey home page Code Monkey logo

declarative-dataflow's Introduction

Declarative Dataflow

A reactive query engine built on differential dataflow.

crates

Documentation

Features

Interactive use: Declarative accepts queries expressed in a Datalog-inspired binding language and turns them into differential dataflows dynamically and at runtime. This works equally well as a library embedded into a specific application or as a standalone service (e.g. via the included WebSocket server).

Reactive relational queries: Declarative provides a relational query language, with full support for traditional binary joins, worst-case optimal n-way joins, antijoins, various aggregates, predicates, unions, and projections. Queries are made up of rules, which can depend on each other (or recursively on themselves). Query results are updated incrementally as inputs change.

[WIP] Reactive GraphQL queries: Declarative also comes with built-in support for GraphQL-like queries, for a more document-oriented usage model.

Pluggable sinks and sources: Declarative can be extended to read data from and write results back to external systems, such as Kafka or Datomic, as well as static sources such as csv files.

Pluggable frontends: Languages such as Datalog and SQL can be easily implemented on top of Declarative. Well, maybe not easily, but easier than without. A Datalog frontend is provided in Clojure(Script).

Thanks to Differential Dataflow, all these capabilities are provided within the dataflow model and can thus be scaled out to multiple independent workers. Declarative is less efficient and much more opinionated than hand-written Differential Dataflow. In particular, it enforces a fully-normalized, RDF-like data model heavily inspired by systems like Datomic or LogicBlox. Other than that, Declarative is just Differential Dataflow under the hood and can happily co-exist and interact with static, handwritten dataflows.

Included in this repository is the library itself, a server, and a cli.

Build

The library is built using cargo. A sufficiently up-to-date Rust toolchain is enforced via the rust-toolchain file included in this repository.

Declarative Dataflow makes use of the log crate. Logging at a specific level can be enabled by setting the RUST_LOG environment variable to RUST_LOG=declarative_dataflow=<level>.

Documentation

Crate documentation available on crates.io.

Important architectural decisions are documented in the docs/adr/ sub-directory.

Documentation for this package can be built via cargo doc --no-deps and viewed in a browser via cargo doc --no-deps --open. Please refer to declarative_dataflow::plan::Plan for documentation on the available operators. The tests/ directory contains usage examples.

Clients

Declarative Dataflow servers accept a lower-level relational query plan representation, that is meant to be somewhat frontend agnosti. However query plans are rather cumbersome to write manually and do not map to any interesting, higher-level semantics. Currently we therefore provide a Datalog front end written in Clojure.

Please refer to the documentation for an overview of the supported query plans.

Further Reading / Watching

A post on the high-level motivation for this project.

[video] Reactive Datalog For Datomic, recorded at Clojure/conj 2018.

[video] 3DF: Reactive Datalog For Datomic, recorded at :clojureD 2019.

The Clockworks blog has a number of posts on Declarative.

declarative-dataflow's People

Contributors

comnik avatar frankmcsherry avatar li1 avatar

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.