Reads row-level changes from CockroachDB then sends them to a message queue!

This app utilises Core changefeeds "which stream row-level changes to the client indefinitely until the underlying connection is closed or the changefeed is canceled".

• RabbitMQ

• CockroachDB

1. The cursor store is read to check if a cursor is already stored.
2. A changefeed is created with the EXPERIMENTAL CHANGEFEED FOR {table} expression with the following options:
   1. frequency - this is how often a new cursor should be returned from cockroachdb.
   2. cursor - this is the earliest time a change should be returned.
3. Response rows are iterated to find rows that describe a change and rows which have the latest resolved cursor.
4. If the row describes a change:
   1. The arguments are parsed into the table, key and value.
   2. These are wrapped into a new struct and encoded into JSON.
   3. The new JSON payload is sent to the message queue.
5. If the row is a new resolved cursor:
   1. The value argument is parsed as the others should be null.
   2. The resolved JSON field is pulled from the value response.
   3. This is inserted/updated in the cursor store.

SQL:

CREATE TABLE foo (a INT PRIMARY KEY, b int);
INSERT INTO foo VALUES (54);

Published message:

{"table":"foo","key":"[54]","value":{"after": {"a": 54, "b": null}}}

Tested on Rust version 1.46.0-nightly (346aec9b0 2020-07-11).

docker run duccos/crdb-changefeed-publisher:latest

Example deployments can be found in the .kube directory.

1. git clone [email protected]:ducc/crdb-changefeed-publisher.git
2. cd crdb-changefeed-publisher
3. cargo build --release
4. {environment vars} target/release/crdb-changefeed-publisher

Prometheus metrics are exposed on the PROMETHEUS_ADDR env var address at /metrics.

Licensed under MIT. See the LICENSE file in the repository root for the full text. This is a third party application and is not affiliated with CockroachDB. It is not a direct replacement for the Enterprise Changefeeds offering.

Feel free to submit a merge request! Your changes MUST be submitted under the MIT license.