nicusx / event-sourcing-kafka-streams Goto Github PK

Add a Docker-Compose stack with

• Kafka
• ZooKeeper
• MySQL

Create the db schema and set db user directly in the container

Add support for Schema Registry.
Add Schema Registry to the Docker Compose stack (depends on #1)

Does this require replacing Avro4S? Probably.

Build project on TravisCI

At the moment, the implementation assumes there is a single instance of each service: Command Handler, List Projector, Web Server

Allow scaling out each service type.

This probably requires specifying the consumer group.
It also probably requires modifying the web service: at the moment POST, /execute/{invoiceId} listen to the command result topic waiting for the result corresponding to the commandId.
This assumes a single instance of the service or a single-partition topic for Command Results.
One possible simple solution may be using a different consumer group per service instance. This way all instances will consume all partitions.

All commands except CreateInvoice should fail when the invoice does not exist.

It should generate a Failure CommandResult message.

Depends on #16

Document the endpoints

A CreateInboce command should fail when an Invoice with the specified invoiceId already exists, even when no expected version is set

It should generate a Failure CommandResult message.

Depends on #16

A PayInvoice command should fail when the target invoice is already payd

Externalise the whole configuration in config files (as opposed reading env variables from a shared class org.amitayh.invoices.common.Config._)
Use separate config files per process (not a single one in common)

Maybe use https://pureconfig.github.io/docs/

Set processing.guarantee to exactly_once

Add tests for Kafka Stream Topologies
See https://kafka.apache.org/21/documentation/streams/developer-guide/testing.html

Generate AVRO Objects from schema (rather than the other way around).

CommandResult SerDe fails when serialising a CommandResult with a Failure outcome

Provide the ability to rebuild the snapshots topic from scratch, consuming all events.
This should happen on demands only and not while the command handler is running.
Probably a separate application.

A dumb implementation might assume the snapshot topic is empty end re-consume all events from the beginning of the time.
A smarter implementation should consume both snapshots and events streams, only applying events starting from the version of the latest snapshot (+1)
(remember: the Event version is the version of the Snapshot it is supposed to be applied to)

Split application in separate deployable components:

1. Web (API)
2. Command Handler
3. List Projector

Currently, CommandHandler update the stateful Snapshot table as a side effect of processing commands.
This sounds quite wrong and does not provide the ability to rebuild the snapshots from scratch and make sure the snapshot stream is consistent with the stateful Snapshot table.

The actual process should be the following:

1. Applying a Command to the current state generates a Command Result: either a failure or Event(s). Both CommandResults and Events are materialised as Topics.
2. Applying events to the current Snapshot generate the new Snapshot.
3. The new Snapshot update the stateful Snapshot table and is also materialised as a Topic

This is the same process described here