These instructions assume you have (and know how to work) python3.6, Docker

1. Docker containers
2. Python 3.6
3. Django
4. Postgres
5. S3 buckets/objects
6. RabbitMQ
7. Celery workers

1. Client POST a GeneParser object to API Server.
   1. API Server contacts S3 and retrieves a presigned URL for upload.
   2. API Server stores it in the GeneParser object and returns to client.
2. Client PUT a file in the S3 presigned URL.
3. Client POST to API Server that the file was uploaded.
4. API Server creates AsyncJob and puts a task in the RabbitMQ.
5. Worker pulls job from RabbitMQ and begins processing.
   1. Worker downloads file in chunks from S3 and processes each chunk separately.
   2. Once the Worker finds a DNA sequence it stores it in Postgres.
   3. When file is complete the worker builds a JSON from the stored DNA sequences and sends it to S3.
6. Client polls AsyncJob progress until completed.
7. Client GET GeneParser
   1. API Server contacts S3 and retrieves a presigned URL for download.
8. Client downloads file using presigned URL.

1. Create terminal and navigate to project folder.
2. docker network create genobase_default
3. docker-compose -f data.yml up
   1. This will bring up Postgres, Minio and RabbitMQ
4. Wait for the database to initialize...
5. Create terminal and navigate to project folder.
6. docker-compose -f processors.yml up
   1. This will bring up the API server, Celery workers and a schema migrations server.
7. Create terminal and navigate to project folder.
   1. pip install requests: Needed for the test script
   2. Run the test script: python tests/test_full_flow.py
   3. Navigate to ./downloads/ and see the results.

1. ./gene_parser/tasks.py: The processor background job.
2. ./gene_parser/parsers.py: The file text parsers.
3. ./downloads/: Where the resulted parsers are.
4. .tests/test_full_flow.py: The test script.

1. Since objects are stored in the DB it's simple to create an API to retrieve them in different manners.
2. Code assumes genes are not repeating in the files.
3. Improvement idea: Shard file to multiple workers.
4. Improvement idea: Use S3 bucket notifications instead of the file_uploaded endpoint.
5. Improvement idea: Store file in S3 in parts instead of whole.
6. I'm using python manage.py runserver as the server for demo simplicity.
7. No Authentication layer or any security for simplicity.
8. If you want to reset everything just delete the ./data folder.