Code Monkey home page Code Monkey logo

datastax-bulkloader-writer-example's Introduction

Data Loader and Writer example

This demo creates the sstable files and loads them through jmx to a cassandra cluster. It then loads transactions into a file of a certain size. The default is to load 1 million records and create files of 100,000 transactions in each file.

The files are routed to different files based on the transaction type (Debit/Credit). This could be changed to be source feed or bank provider for example.

Running the demo

To run this code, you need to have your cluster 'cassandra.yaml' and 'log4j-tools.properties' in the 'src/main/resources' directory.

You will need a java 7 runtime along with maven 3 to run this demo. Start DSE 4.0.X or a cassandra 2.0.5 or greater instance on your local machine. This demo just runs as a standalone process on the localhost.

This demo uses quite a lot of memory so it is worth setting the MAVEN_OPTS to run maven with more memory

export MAVEN_OPTS=-Xmx512M

Schema Setup

Note : This will drop the keyspace "datastax_bulkload_writer_demo" and create a new one. All existing data will be lost.

To create the a single node cluster with replication factor of 1 for standard localhost setup, run the following

mvn clean compile exec:java -Dexec.mainClass="com.datastax.demo.SchemaSetup"

To run the bulk loader, this defaults to 1 million rows.

mvn clean compile exec:java -Dexec.mainClass="com.datastax.bulkloader.Main" 

To run it other settings for no of rows, jmx host and port

mvn clean compile exec:java -Dexec.mainClass="com.datastax.bulkloader.Main"  -DnoOfRows=2000000 -Djmxhost=cassandra1 

To run the report writer to fill a file with a batchSize of transactions

mvn clean compile exec:java -Dexec.mainClass="com.datastax.reports.ReportWriterImpl"

The default is 100,000 transactions per file, to change this use the batchSize property

mvn clean compile exec:java -Dexec.mainClass="com.datastax.reports.ReportWriterImpl" -DbatchSize=15000

To remove the tables and the schema, run the following.

mvn clean compile exec:java -Dexec.mainClass="com.datastax.demo.SchemaTeardown"

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.