zedr0n / zes Goto Github PK

Redis Streams available from 5.0 can be used as a backend for event store

With the changes from #9, the schema is now stitched explicitly and with the corresponding HotChocolate upgrade the subscriptions should start working now.

HotChocolate server has changed API from version 10 so the code has to be updated to the new specifications

Currently ZES supports SQLStreamStore and EventStore as backend. The common code was separated in EventStoreBase but several issues are outstanding:

• #7
• #5
• #4
• #16

After recent branch changes parent is only set if target branch version is ahead of immediate parent, need to check what happens if version is ahead of just some of the ancestors

Use a separate class for metadata in the event to allow separate serialization and separation of concern.

Potentially, there's also mutable metadata ( version, stream, timeline ) and read-only metadata

For the events and commands resulting from others, e.g. caused by sagas, depth is the distance from originating command which can be tracked or explicitly added to event metadata

• #14

Make sure that when an invalid state is detected, the command to roll back is identified via the ancestor id of the event rather than just the explicit command id

The framework has a GraphQL interface which can be used to interface with Python

• Python graphql client
• #11
• #9

Introduce the concept of t- time which describes the state just before t so doesn't includes the events/commands with exact time t.

Then to get events resulting from the command can be done by replaying command at its timestamp- branch

DragonflyDB is a multi-threaded replacement for Redis which promises 25x throughput. This is fully api-compatible with Redis so could be tested as a drop-in replacement.

Objective:
To ensure traceability and establish causal relationships between events in our event-sourced system, we aim to introduce a hashing mechanism. This will aid in identifying the impact and lineage of events, especially within sagas and aggregates.

Background:
Our current event-sourced system relies heavily on sagas, which are event-sourced state machines. These sagas interact with multiple aggregates, often generating commands based on events they listen to. This interplay makes tracing the lineage and impact of events critical for both system correctness and debuggability.

Proposed Solution:

1.	Event Hashing:
•	Upon the creation of an event, compute a hash of the aggregate or saga’s state. This will serve as a compact representation of the state at that point in time.
•	Store the hash as part of the event’s metadata.
•	Utilize a robust and efficient hashing algorithm suitable for our system’s needs, ensuring minimal collisions.
2.	Correlation and Causation:
•	Maintain a correlation ID in the event metadata to trace events that belong to the same overarching transaction or process.
•	Introduce a causation ID ( issue #13 ) to link commands and the resulting events, enabling a clear understanding of the cause-effect chain.
3.	Saga Design:
•	Ensure sagas, as state machines, transition between states in response to events. The hash should encompass these state transitions.
•	Sagas listening to events should maintain a correlation with the originating aggregate’s event, preserving the event lineage.

Expected Outcome:

•	Enhanced traceability and understanding of the flow and impact of events across the system.
•	Simplified debugging and anomaly detection by following the causal chain of events.

📢 Proposal:

Integrate a “Causation ID” mechanism into our event-sourced system to ensure clear traceability of events, especially those triggered indirectly through sagas.

🎯 Objective:

•	Provide a direct link from a resulting event back to the event, command, or decision that caused it.
•	Enhance the debugging and monitoring capabilities by illuminating the lineage of events.

🔍 Context:

In our current design:

•	Events don’t trigger other events directly.
•	Instead, they activate sagas which issue commands leading to new events.

This indirect chaining, while flexible, can make tracing the sequence of events challenging. Introducing a Causation ID fills this traceability gap.

🚀 Implementation:

1.	Event to Saga:
•	When a saga processes an event and generates a command, assign the event’s ID as the Causation ID for the command.
2.	Command to Event:
•	Upon processing a command and producing a new event, transfer the Causation ID from the command to this new event.
3.	Metadata Storage:
•	Store the Causation ID within the event’s metadata, ensuring quick access and minimizing impact on the event payload.

⚖️ Pros & Cons:

Pros:

•	Facilitates a clear visualization of the event flow.
•	Assists in troubleshooting and understanding event sequences.
•	Minimal overhead with significant value.

Cons:

•	Needs careful integration with existing mechanisms like hashing for causal inference.
•	Might require adjustments in our logging and monitoring setup to leverage fully.

Explicit sql implementation allows to customize the storage of the metadata

• #16

Retroactive operations with large-ish streams are proving to be very slow irrespective of backend.

• #17

Add support for explicit SQL backend

Check if command has already been processed by querying the command log before processing it

• #24

Background:
Currently, our event-sourced system utilizes copies of aggregate events within sagas. While this approach ensures that sagas have all the information they need, it introduces redundancy and potential synchronization issues between aggregates and sagas. A more streamlined approach would involve using distinct saga-specific events, which would be clearer in intent and reduce duplication.

Objective:
Transition from using copies of aggregate events in sagas to creating and using saga-specific events. This should enhance clarity, reduce redundancy, and prevent potential desynchronization issues.

The command is currently marked as complete when the corresponding handler has finished, i.e. the direct resulting events have been saved.

The full command completion is when all the sagas have finished reacting to the cascading events and side effects.

Include the command type in the id e.g. CreateRoot-{GUID}

The read functionality of the event store is implemented as an RX observable. Given that TPL Dataflows are extensively used in the rest of the codebase, is that justified?