Code Monkey home page Code Monkey logo

bankinabox's Introduction

BankInABox

This is a simple implementation of a Retail Banking application, intended to show features of Hazelcast IMDG and Hazelcast Jet being used cooperatively.

The initial implementation is focused on Fraud Detection. In IMDG, an Executor-based RuleEngine is used to execute RuleSets; each RuleSet consists of a set of rules and then an aggregation of the rule results into an overall pass/fail score. Two rulesets are currently in place; one for Location based rules and one for Merchant based rules.

One-time setup:

You'll need to have the following dependencies installed in order to run this demo.

  • Java JDK 11 or later
  • maven
  • docker

Once you have the project installed (cloned from github), you'll need to build it.

mvn clean package

(Or use your favorite IDE to build the top-level project).

Note that the project requires JDK 11 or later to build.

Start MariaDB (Docker Image)

When running, the demo operates all in-memory, but the data is originally loaded from a MariaDB database. To simplify running the demo, there is a Docker image that includes MariaDB and test data for running the demo. To start MariaDB running in a Docker image, do the following:

For MacOS, this can be done as follows

cd mariadb
make up
lsof -i : 3306

The last line is optional but a good check -- you should see that the docker copy of MariaDB is listening on port 3306. (For compatibility, MariaDB identifies itself as MySQL on the output)

Start IMDG Cluster:

First, start one or more IMDG Cluster nodes using the script runImdg.sh

./runImdg.sh

I generally start 3 nodes. Earlier versions of the demo required this because some features of the CP Subsystem were used, but the current version can run with a single IMDG if desired.

Start Jet Cluster

Start one or more Jet Cluster nodes using the script runJet.sh

./runJet.sh

A single node here should be sufficient.

Start Grafana

You should also start the Grafana dashboard in order to observe the output generated by the demo. From the top-level BankInABox directory

make up

You can then point a browser to https://localhost:80 and log in with the default username and password.

First time setup

The first time you start Grafana you'll need to import the customized BankInABox dashboard. Select the Grafana icon (Upper left corner of the screen), the Dashboards menu, and select Import from the pop-out submenu.

In the import dialog, click the box that says Import JSON file, and navigate to the dashboard file location

[BankInABox home]/grafana/Bank In A Box-dashboard.json

Once the dashboard is imported, you can go to the Grafana home page and then select the dashboard from the list of installed dashboards. All the gauges will be reading zero until the demo begins.

Start the demo (Launcher script)

Finally, start the demo itself by running the following script in the top-level BankInABox directory

./runLauncher.sh

The launcher script runs the main Launcher class in the demo application, which does all of the following:

  • Triggers a preload of the Merchant and Account info from the database into the data grid
  • Sets a listener on the preAuth map to detect incoming transactions. The listener forwards trasactions to each RuleSet Executor by writing the transaction to a ruleset specific Queue.
  • Creates a Jet pipeline to monitor the incoming transaction stream and aggregate the average transaction amount by merchant, and submits a job to the Jet cluster to execute the pipeline.
  • Starts executors for each RuleSet (current demo has two)
  • Starts an executor to aggregate completed RuleSet results
  • Starts a periodic task (5 second interval) that writes data to the Graphite sink used by the Grafana dashboard to show results.
  • Loads data from the database Transaction table into the preAuth map; this then triggers the listener and Jet pipeline to begin processing.

The Grafana dashboard you started earlier can be used to monitor the transaction results; you can see the transaction volume, fraud rate, and average latency per transaction. (Note that there is a warm-up time for the demo due to the initial database load; the latency will start high but then drop down as the maps get populated.)

Hazelcast Management Center can be used to monitor the maps and watch how data moves between the maps. (Generated transactions go into the preAuth map; once they have been processed they will move to either approved, rejectedForCredit, or rejectedForFraud).

Jet Management Center can be used to monitor the Jet pipeline job. The pipeline watches the transaction stream and calculates updated average transaction amounts per merchant. The Jet Management Center startup script needs to be modified to connect to port 5710 rather than default 5701, so that Jet and IMDG can run on the same server without collision.

bankinabox's People

Contributors

7erry avatar myawn avatar myawnhc avatar neilstevenson avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar

Forkers

rimogitz cckmit

bankinabox's Issues

SEVERE error encountered on Jet instance

Ran the demo as documented with one exception - 2 IMDG members instead of 3, and ran into the below error after 7 mins run:

SEVERE: [localhost]:5710 [JetInABox] [3.1] Execution of job 'AdjustMerchantTransactionAverage', execution 245a-d151-63fc-9ae2 failed after 408,673 ms
com.hazelcast.jet.JetException: Exception in ProcessorTasklet{AdjustMerchantTransactionAverage/Draw Transactions from preAuth map#0}: com.hazelcast.jet.JetException: Null key from key extractor, edge: between("Draw Transactions from preAuth map", "Aggregate average transaction amount by merchant-prepare").partitioned(?)
	at com.hazelcast.jet.impl.execution.TaskletExecutionService$CooperativeWorker.runTasklet(TaskletExecutionService.java:315)
	at java.base/java.util.concurrent.CopyOnWriteArrayList.forEach(CopyOnWriteArrayList.java:803)
	at com.hazelcast.jet.impl.execution.TaskletExecutionService$CooperativeWorker.run(TaskletExecutionService.java:289)
	at java.base/java.lang.Thread.run(Thread.java:834)
	at ------ submitted from ------.(Unknown Source)
	at com.hazelcast.spi.impl.operationservice.impl.InvocationFuture.resolve(InvocationFuture.java:126)
	at com.hazelcast.spi.impl.AbstractInvocationFuture$1.run(AbstractInvocationFuture.java:251)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:834)
	at com.hazelcast.util.executor.HazelcastManagedThread.executeRun(HazelcastManagedThread.java:64)
	at com.hazelcast.util.executor.HazelcastManagedThread.run(HazelcastManagedThread.java:80)
Caused by: com.hazelcast.jet.JetException: Null key from key extractor, edge: between("Draw Transactions from preAuth map", "Aggregate average transaction amount by merchant-prepare").partitioned(?)
	at com.hazelcast.jet.core.Edge$KeyPartitioner.getPartition(Edge.java:588)
	at com.hazelcast.jet.impl.execution.OutboundCollector$Partitioned.offer(OutboundCollector.java:193)
	at com.hazelcast.jet.impl.execution.OutboxImpl.doOffer(OutboxImpl.java:241)
	at com.hazelcast.jet.impl.execution.OutboxImpl.offerInternal(OutboxImpl.java:141)
	at com.hazelcast.jet.impl.execution.OutboxImpl.offer(OutboxImpl.java:93)
	at com.hazelcast.jet.core.AbstractProcessor.tryEmit(AbstractProcessor.java:310)
	at com.hazelcast.jet.core.AbstractProcessor.emitFromTraverser(AbstractProcessor.java:410)
	at com.hazelcast.jet.core.AbstractProcessor.emitFromTraverser(AbstractProcessor.java:423)
	at com.hazelcast.jet.impl.connector.StreamEventJournalP.emitResultSet(StreamEventJournalP.java:209)
	at com.hazelcast.jet.impl.connector.StreamEventJournalP.complete(StreamEventJournalP.java:194)
	at com.hazelcast.jet.impl.execution.ProcessorTasklet.stateMachineStep(ProcessorTasklet.java:367)
	at com.hazelcast.jet.impl.execution.ProcessorTasklet.call(ProcessorTasklet.java:232)
	at com.hazelcast.jet.impl.execution.TaskletExecutionService$CooperativeWorker.runTasklet(TaskletExecutionService.java:308)
	at java.base/java.util.concurrent.CopyOnWriteArrayList.forEach(CopyOnWriteArrayList.java:803)
	at com.hazelcast.jet.impl.execution.TaskletExecutionService$CooperativeWorker.run(TaskletExecutionService.java:289)
	at java.base/java.lang.Thread.run(Thread.java:834)

Integrate with ML/AI

Create a RuleSet and rules that utilize a machine learning model to evaluate transactions and flag potential fraud.

Change minimum Java version support to 8

Currently the minimum Java version required is 11, which does not align with Hazelcast's offering.
Additionally, not a large developer population is on 11 yet and forcing them is likely to be discouraging.
This should be brought down to Java 8.

(consider) replace multiple queues with single topic

Currently, the preAuth map listener writes the same transaction to multiple queues in order to pass it to different RuleExecutors for processing. This is an expensive duplication - maybe not too bad if executors are keeping up, but will quickly become an issue if executors fall behind.

A pub/sub model using ReliableTopic seems preferable.

The current RuleExecutor is built around CompletableFuture and works well with the IQueue.take mechanism for input; it's not clear how to adapt this to the event-driven onMessage style of a topic.

Variable rate, on-demand transaction ingest

Current design: Transaction generator runs concurrently with the processing, generating random transactions that are written to the preAuth map. So transaction generation is competing for the same CPU cores that the processing needs.

Preferred design: Pregenerate transactions and store them (database or whatever); have an ingest mechanism that can vary in rate to drive different loads for testing. Rate should be variable to simulate spikes as expected in real-world workloads.

Support multiple run modes

Allow multiple run modes - Demo and Benchmark

Various things will work differently depending on run mode.

  • Demo mode is meant to run continuously (for example, all day at a trade show)

  • Benchmark mode is meant to run for a fixed length and collect statistics that can be used to compare different design alternatives.

  • Demo mode needs to be more concerned about out of memory issues and manage the working set carefully via map eviction, etc.

  • Demo mode supports a Grafana dashboard that isn't needed for benchmark

Grafana running empty

Imported dashboard into Grafana and started the demo. There is no activity in Grafana and all dials are sitting idle. On the other hand, Management Centre shows activity.

See attached screenshots of both Grafana and mancentre from the same time.

Screen Shot 2020-03-03 at 9 33 34 am

Screen Shot 2020-03-03 at 9 33 50 am

Replace properties with yaml, json, or xml

Currently configuration is done through a properties file, which is an antiquated design.

Change to use yaml (probably best) or json (or xml but probably distant third choice) for configuration.

Jet should use client-server mode

Jet should be running in client-server mode, but was throwing an exception about not finding the service so I've reverted to embedded for now.

There is also an issue with Jet running client/server in Kubernetes that is fixed in 4.0, so we may leave this embedded until 4.0 or a fix for that issue is added to 3.12.x.

Extend use of IdentifiedDataSerializable

Objects such as Account are IdentifiedDataSerializable

Others such as RuleSetExecutor and CreditLimitCheck are java.io.Serializable

All should ideally be IdentifiedDataSerializable, or at least those that are performance impacting

Add location-based rules

Add a ruleset and rules showing location based fraud detection.

Currently plan is to use geohash to represent various locations (merchant location, customer home location, mobile device location)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.