Create and enable end-to-end test for the batch mode. about fhir-data-pipes HOT 5 CLOSED

@pmanko re. testing the streaming version, what I was thinking about was much simpler, i.e., start from a set of events on the server side (OpenMRS) and a "clean history" on the client side (our pipeline code). Clean history means that the streaming pipeline will read the events and fetch the corresponding records (e.g., by clearing the DB tables for the Atom Feed client or the event index file of Debezium). IOW, imagine we create a few events in OMRS and then freeze the DB to be used for all subsequent tests. Each test will start from scratch on the client side and read these events from OMRS server.

Also thanks for the pointer.

from fhir-data-pipes.

For testing the streaming modules, we have to trigger the creation and update of clinical data on the OpenMRS side.

I've previously done something similar in Postman using the OpenMRS Rest API, but I think Create/Update is now supported for many resources in the FHIR2 module, so we could use FHIR requests to trigger the streaming side.

Also, @bashir2 do you have an approach in mind to testing? This account might be useful: https://engineering.zalando.com/posts/2019/02/end-to-end-microservices.html#:~:text=End%20to%20end%20testing%20is,of%20them%20can%20get%20tricky.

from fhir-data-pipes.

Here is some information on how a simple version of this can work: There are three main pieces (1) the input, i.e., OpenMRS, (2) the pipeline to read FHIR resources from the input and do transformations, (3) the output. For all scenarios, we need simple scripts that bring up the input server, run the pipeline and validate the output. This should be run as part of Travis on each PR.

1. The input is OpenMRS for which we rely on what is done for issue #22, i.e., docker images for OpenMRS and MySQL with some test data pre-loaded. Note this is blocked by issue #38 right now.

2. There are three modes for the pipeline: batch, atom-feed based streaming, and debezium based streaming.

3. The output is either Parquet files or a target FHIR store.

We don't need to cover all combinations.

To test the streaming mode, we can create the events in the DB and save them in the docker image as the initial state. Each time the streaming pipeline runs in the test environment, it reads the events it has not processed (starting from a fake offset in the past) and create the output.

from fhir-data-pipes.

@jecihjoy I am changing the scope of this issue to be for the batch mode only. I will create a separate one for the streaming mode (seems @gitcliff has free cycles to work on that). Please close this once PR #56 is submitted (and thanks).

from fhir-data-pipes.

Fixed by #56

from fhir-data-pipes.