fpj / eo-ingestion Goto Github PK

Implement a simple generator for JSON objects.

Add a test case that aborts part way into writing events from a file. Currently, the test enabled with duringTxnFlag aborts after all events have been written but before commit:

https://github.com/fpj/eo-ingestion/blob/master/src/test/java/io/pravega/eoi/PravegaSynchronizedWriterTest.java#L79

This is a good test. However, another interesting test would be aborting after just a few of the events in a transaction have been written. Specifically, I mean aborting part way in the loop that begins here:

https://github.com/fpj/eo-ingestion/blob/master/src/main/java/io/pravega/eoi/PravegaSynchronizedWriter.java#L232

Add top-level comments to PravegaWriter and PravegaTxnWriter indicating that these are simplified implementations that are missing certain features.

The class ExactlyOnceIngestionSynchronizer is difficult to read and follow the execution flow. It would be nice to reorganize it and/or add plenty of comments so that readers know what is going on. This will likely be a reader's first time reading about StateSynchronizer.

Implement an example in which the source isn't a set of files, but another Pravega stream instead.

The current code supports restart cases, but not cases in which it is possible to have concurrent writers, either because there are multiple writers ingesting data or because of a zombie writer.

It is important to document the failure scenarios that the code covers. it is important to argue that it is correct.