Changestream sources object-level change events from a MySQL Replication Master (configured for row-based replication) by streaming the MySQL binlog and transforming change events into JSON strings that can be published anywhere.

Currently, Amazon Simple Queuing Service (SQS), Amazon Simple Notification Service and Amazon S3 are supported with optional client-side message encryption via AES.

• Documentation
• Getting Started
  • Requirements
  • Setup
  • Building
• Contributing
  • Testing
  • ScalaStyle
• License
• Authors
• Inspiration

• Why SNS+SQS?
• How Do I Consume My Events?
• The Stack
• Architecture
• Binlog Event Routing
• IntelliJ
• Configuration

• Scala 2.11
• Akka 2.5
• SBT 0.13
• MySQL 5.7 (5.5+ supported)

First, start MySQL with

docker-compose up

(or, for the hard way without Docker, follow the bespoke setup instructions).

You will then have access to a MySQL instance on localhost port 3306 with a root password of "password", and Changestream configured to listen for events.

Changestream is available on Docker Hub.

Then build and run Changestream using the sbt CLI:

$ sbt run

Note: The Dockerfile will be written to target/docker/Dockerfile, and the docker image will be added to your local docker repo.

$ sbt docker

$ sbt debian:packageBin

$ sbt package

Changestream is 100% unit and integration tested, with end-to-end testing on CircleCI with MySQL and Amazon SNS.

In order to run tests locally you need to configure test.conf with a working MySQL connection configuration.

Make sure that your test user has permissions to create and drop databases and tables. By default, Changestream uses the changestream_test table for tests.

To run the integration tests, as well as the unit tests on the SqsActor and SnsActor, you also need to ensure that you have configured your AWS tokens to be available from the environment, and that your AWS tokens have access to create and add messages to queues (SQS) and create and publish to topics (SNS).

AWS_ACCESS_KEY_ID=<access_key_id>
AWS_SECRET_ACCESS_KEY=<secret_access_key>
AWS_REGION=<your_region>
AWS_S3_BUCKET=<your_bucket>
AWS_S3_KEY_PREFIX=<some_key_prefix>

Once that's all set up, you can either run

$ sbt test

from the command line, or run the test command in IntelliJ (with the Scala and SBT plugins installed).

Changestream uses ScalaStyle for code linting--it is not automatic. Before submitting a PR, please take a look at the output of the sbt scalastyle command and correct any issues you find.

Changestream is available under the MIT License.

Peter Fry (@racerpeter) and the Mavengineering team.

Changestream was inspired in a large part by mardambey/mypipe by Hisham Mardam-Bey. The first proof-of-concept for changestream was a fork of mypipe - thank you Hisham.