The MVP will cover just a use case and it will be evolved later to the complete app.

The docker containers used for testing must be based on docker images tagged for the specific commit to be tested.

In the laboratory we have deployed the service in the cluster using:

terraform apply
terraform output | grep '"' | awk -F '"' 'BEGIN {i=1; print "[cluster]"} {print "node0" i++ " ansible_host="$2}' > ../ansible/swarm/hosts.ini

ansible-playbook -i hosts.ini -u ubuntu create_swarm.yml

scp docker-compose-swarm.yaml [email protected]:
ssh [email protected]
docker stack deploy --compose-file docker-compose-swarm.yaml covid19

The goal of this exercise is to automate using an ansible playbook the last three steps:

scp docker-compose-swarm.yaml [email protected]:
ssh [email protected]
docker stack deploy --compose-file docker-compose-swarm.yaml covid19

so we can deploy the service using:

ansible-playbook -i hosts.ini -u ubuntu deploy_service.yml

We don't need to convert to CSV to get the dataframe. It is built from JSON directly, as shown already with the code.

The CSV could be used to fetch the database, but the database could be also fetched easily from the dataframe.

Also, the source of the diagram must be included to make easier to modify it.

Description

We have decided that the application will manage the covid19 serology tests results in order to get insights to understand better the covid19 pandemic.

The main user of our application will be data journalists in order to analyze and filter the data sets for the purpose of creating a news story, in our case in the covide19 pandemic field.

For doing it the main steps are:

• Describe our data sources with the test results: our only data source will be https://open.fda.gov/apis/device/covid19serology/
• Describe the main requirements (#8) for the application: answers for questions useful for creating news stories.
• Define the use cases detected (#9)
• Define the architecture used to implement the use cases (#10)

Task acceptance criteria

• The README.md of the project reflects the goals of the project

https://docs.github.com/en/actions/publishing-packages/publishing-docker-images

Implement a pipeline with the steps:

• Download the source code
• Pass the tests
• Generate the docker image
• Upload the docker image

In the future the pipeline must also do the deployment (to have CD)

Once we have created the infra for the docker swarm cluster, we need to configure the cluster, adding all the nodes to the cluster.

To get the ip addresses of all the nodes in the cluster and create an ansible inventory with it:

terraform output | grep '"' | awk -F '"' 'BEGIN {i=1; print "[cluster]"} {print "node0" i++ " ansible_host="$2}' > ../ansible/swarm/hosts.ini

Once we have the inventory, we just need to follow the steps documented in the ansible folder

After the refinement of this tasks the conclusions:

• Unzip and read the JSON file into python dictionaries
• Convert Python dictionaries to Dataframes
• Offer the dataframes API for querying the data for getting insights

Now, time to wait for PRs implementing it!

Answers for questions useful for creating news stories.

The data collected is described at: https://open.fda.gov/apis/device/covid19serology/

Some ideas for the questions:

• How many tests were conducted and the evolution in time aggregated
• Global number and evolution of tests per:
  • Manufacturer of the test
  • Results of the test
  • Veracity of the results of the test

In general the goal is to understand the performance (detect the antibodies) of this tests. The antibodies only appear some days after the infection.

• Plan
  • Implement a basic use case of our covid19serology tests data management application for journalists
• Create
  • How to capture the data from OpenFDA #13
  • How to parse the data #14
  • How to implement the API REST to offer the data to the portal #15
  • How to implement the portal which consumes the API REST and provides de insights to the data journalist #16
• Verify
  • Add basic testing #17
• Package
  • Create a docker image to deploy the software #18
• Release
  • Tag the docker image to have release control over it #19
  • Publish it in a docker repository #20
  • Create the infra needed in AWS to deploy it #21
• Configuration
  • Define configuration params to provide the service #22
• Monitor
  • Use Cloud Watch to monitor the deployed service #23
  • Analyze the use of sentry #24
  • Analyze the use of Grafana #25

The PoC will cover the DevOps infinite loop

Publish the docker image to https://hub.docker.com/

With docker run and docker-composeall the containers are started in a single host.

In order to distribute the containers to different hosts, but working together, we need an orchestrator.

In the first example, let's do it with docker swarm, included as a module in docker, and easy to use.

And after it, let's use the possibility of using a docker-compose file to deploy the service to a swarm:

https://docs.docker.com/engine/swarm/stack-deploy/

The data is downloaded from: https://open.fda.gov/apis/device/covid19serology/ using the API.

But it can be downloaded totally from https://download.open.fda.gov/device/covid19serology/device-covid19serology-0001-of-0001.json.zip