Code Monkey home page Code Monkey logo

dada's Introduction

DADA

I've tested building on various i386 and x86_64 Java/Linux platforms. You will need a recent JDK and Maven (and probably JAVA_HOME and MAVEN_HOME set).

For a demo of DADA please follow the instructions below. The demo domain is whale sightings (I am fascinated by whales), however DADA is generic enough to slice, dice and aggregate pretty much anything in realtime. At the bank at which I currently work, it is being used by Treasury to project 6 month cash trade settlement ladders and highlight late amendments.

Build DADA

git clone https://github.com/JulesGosnell/dada.git
cd dada
mvn clean install

In one shell: Start DADA Whales Demo Server

cd dada/dada-demo
../bin/clj
(load-file "src/main/clojure/org/dada/demo/whales.clj")

In another shell: Start DADA Client GUI

cd dada/dada-swt
../bin/clj
(System/exit 0)
cd ..
./bin/client

You should now be looking at DADA's MetaModel - it's Model of Models.

Alt text

Click on the Whales Model.

Alt text

  • A View of the Whales Model should open (you can open as many as you like)
  • This Model contains a live feed of whale sightings
  • A whale's attributes (except type) may vary from sighting to sighting

Click on the Oceans Model.

Alt text

  • A View of the Oceans Model should open (you can open as many as you like)
  • This Model contains a live feed of Ocean samplings
  • An Ocean's attributes may vary from sample to sample

Click on the WhalesAndOceans Model.

Alt text

  • The WhalesAndOceans Model is derived from the Whales and Oceans Models by a query
  • This query demonstrates DADA's ability to join live streams/Models in real time
  • Each Whale is shown joined to its current Ocean (which may change over time)
  • Whale and Ocean attribute changes are reflected in realtime in the WhalesAndOceans Model

In DADA Whales Demo Server shell/repl:

;; select namespace
(ns org.dada.demo.whales)
;; run query
(? (dunion)(dsum :weight)(dsplit :ocean)(dfrom "Whales"))
  • You should see a number of new Models leap into existence in the MetaModel
  • These Models are derived from the inital Whales and Oceans models via the query above
  • Intermediate Models are maintained so that they may also be viewed

Alt text

Click on the Whales:ocean.sum(:weight).union() Model.

Alt text

  • This Model maintains the ultimate result of the query that you have just entered
  • The Whales have been grouped by Ocean and the weight of Whales in each of these groups summed up.
  • As the weight and ocean attributes of the Whales change over time, so does this Model

Click on the Whales.split(:ocean=arctic) Model.

Alt text

  • This is an intermediate Model contributing to the ultimate Model above.
  • It maintains the real time set of all Whales currently in the Arctic.

Click on the Whales.split(:ocean=arctic).sum(:weight) Model.

Alt text

  • This is a intermediate Model drawing upon another intermediate Model and contributing to the ultimate Model above.
  • It represents the real time sum of the weight of all Whales currently in the Arctic.

DADA is a proof of concept, allowing me to learn about Event Stream Processing in Clojure from the inside, not (yet) production software.

Clojure is an ideal platform on which to implement ESP - A functional approach, sequence comprehension, simple concurrency, persistant containers and immutable record types.

The only serious change that I have had to make to Clojure is enabling the recording of dynamically created classes on the server and their on-demand loading into a client. This allows me to run queries server-side which create new functions and types and then to have these pulled into the running client when needed to view the resultant Models.

dada's People

Contributors

julesgosnell avatar

Stargazers

 avatar  avatar  avatar

Watchers

 avatar  avatar

Forkers

vijayeluri

dada's Issues

Version / Statefullness / Asynchronicity

Models are stateful
Models may therefore receive out-of-order updates and resolve them correctly
Models may therefore be connected in an asynchronous fashion to their inputs.

Views such as Router are stateless - so assuming a Model at either side, can be hooked up asynchronously

However Aggregators maybe stateful - e.g. MAX/AVERAGE - does this mean that they must always be connected synchronously to a stateful component ? If so we cannot have Model->Router->Aggregator..

Interfaces for Stateful/Stateless Sync/Async components need to reflect these properties so that they can not be plugged together in a way that won't (always) work....

add query method to MetaModel

extend MetaModel i/f with e.g. query(String query) method
add a setQueryFn(QueryFn queryFn) to MetaModelImpl
Define an interface QueryFn
etc.

implement AggregatedModelView

SUM, PRODUCT, AVERAGE, MEDIAN, COUNT, MIN, MAX etc - should all be abstractable into Aggregator interface and be injectable from Clojure

start using clojure-maven-plugin

  <plugin>
    <groupId>com.theoryinpractise</groupId>
    <artifactId>clojure-maven-plugin</artifactId>
    <version>1.3.1</version>
    <configuration>
      <clojureOptions>-Xmx128</clojureOptions>
      <warnOnReflection>true</warnOnReflection>
      <namespaces>
        <namespace>org.dada.core</namespace>
        <namespace>!org.dada.demo.*</namespace>
      </namespaces>
    </configuration>
    <executions>
      <execution>
        <id>clojure-compile</id>
        <phase>compile</phase>
        <goals>
          <goal>compile</goal>
        </goals>
      </execution>
    </executions>
  </plugin>

move clj files into src/main/clojure
may need to fix up some ns declaration for core and whales

aggregators MAX() and MIN() are problematic...

If an aggregator is holding e.g. the biggest value that it has seen so far and it then receives and update/deletion which removes this value from the dataset, it will not have another value to fall back on.

If we retain an sorted list of all discrete values seen, we can resolve this, but the aggregator becomes much more expensive.

If we add necessary API for aggregator to request total refresh from its Model, this may result in the very costly refreshing of all intermediate Views...

I'm leaning towards the former solution, but I'll sleep on it.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.