Based on Assignment 1, improve your system as follows

1. If you have not already done so, containerize each of your microservices using Docker.

2. Establish CI/CD systems for each of your microservices, including tests.

3. You should be able to trigger builds and live deployments through GitHub commits to your “dev” and “release” code branches for each service.

4. Use Kubernetes to manage the deployments of your containers

5. Deploy your system on Jetstream using the OpenStack API (Do not use Jetstream’s Atmosphere.)

6. You should also automate/script the creation and management of your VMs and deployments of Kubernetes. A tool like Terraform from HashiCorp may be useful but isn’t required.

7. Using a tool such as JMeter, evaluate the scaling of your system

8. Using JMeter or a similar tool, measure and analyze the performance of your system’s throughput under incrementally increasing loads

9. Test with 1, 3, and 5 replicas (fixed) of each of your services.
   At what point does your system fail?
   What about your system failed?

10. Test your system with elastic resource management (that is, system grows under load, contracts when resources are not needed).

11. Inject failures and demonstrate that your system continues to function with JMeter-created or similar loads.

12. Manually kill a VM running part of your system

13. Your entire system must be deployable on vanilla Jetstream VMs by your peer-reviewers and graders with a single command for provisioning the infrastructure and a single command (a git commit) for deploying your system.

14. They should be able to verify your system works correctly through pre-defined tests, should be able to observe your system operations under JMeter (or other created) loads, and should be able to observe your system’s operations under reasonable failures.

Currently considering:
UI - Vue.js/React.js
API gateway - Rest, gRPC and Thrift
Message handling: Kafka/ RabbitMQ
Services: Python/Java

1. UI / Web Service
2. API gateway
3. User management service
4. Session management service
5. Data Retrieval service
6. Model mock up service
7. Post processing service
8. 3 different languages
9. Different DB for each service if required(No two services should access same Database)
10. User authentication. (User audit)
11. Service calls though message handlers
12. Continuous Integration and deployment system for deploying the services.
13. System independent application.
14. Easily deployable.

Jar and all the executables shouldn't be present in your Github repo.

Now that we have a general idea of the system that is to be built, lets use this discussion thread to put up ideas of how we can use NEXRAD data in our use case for Assignment 1.

1. [Issue] Data retrieval
2. [Implementation] Java wrapper for kafka
3. [Implementation] Kafka-python consumer
4. [Milestone] NSA Hello World
5. [Documentation] Establish contribution instructions

Based on:
https://kafka.apache.org/quickstart

Send message to session-management from user-management after login for login status.

A basic architecture diagram. Let's discuss to make corrections/changes and add more details.

A good candidate:
https://nvie.com/posts/a-successful-git-branching-model/

Develop UI frontend for the project

Referred the below sources:
https://www.youtube.com/watch?v=dQK0VLahrDk
https://www.semantics3.com/blog/a-simplified-guide-to-grpc-in-python-6c4e25f0c506/

Figure out what keys need to be shared in order to collaborate on Jetstream

As per #48

• kafka
• mysql
• user_management
• api_gateway
• data_retrieval
• post_processing
• session_management
• user_interface

Upon trying to log in with wrong credentials, the page does nothing.

A great candidate:
https://airbnb.io/javascript/react/

Create 2 virtual machines on ubuntu using KVM and try out the installation of kubernetes using kubespray.

A good candidate:
https://docs.plone.org/develop/coredev/docs/git.html#making-commits

• https://docs.docker.com/get-started/part2/
• https://docs.docker.com/develop/develop-images/dockerfile_best-practices/
• https://docs.docker.com/compose/gettingstarted/
• https://runnable.com/docker/python/dockerize-your-python-application

Create VM on Jetstream and install openstack client tools

There is an issue with the Dockerfile of postprocessing. Upon trying to build the docker image, I get the following error:

Building postprocessing
Step 1/10 : FROM python:3-onbuild
# Executing 3 build triggers
 ---> Using cache
 ---> Using cache
 ---> 7f1e79e15eaf
Step 2/10 : ADD postprocessing.py /
 ---> 1dc168f020ec
Step 3/10 : ADD pyart-master /
 ---> 7fadd36737d7
Step 4/10 : RUN pip install --upgrade pip
 ---> Running in ff09004fe42d
Collecting pip
  Downloading https://files.pythonhosted.org/packages/54/0c/d01aa759fdc501a58f431eb594a17495f15b88da142ce14b5845662c13f3/pip-20.0.2-py2.py3-none-any.whl (1.4MB)
Installing collected packages: pip
  Found existing installation: pip 10.0.1
    Uninstalling pip-10.0.1:
      Successfully uninstalled pip-10.0.1
Successfully installed pip-20.0.2
Removing intermediate container ff09004fe42d
 ---> b03c2773be49
Step 5/10 : ADD requirements.txt /
 ---> 50691c8e7dbf
Step 6/10 : RUN pip3 install -r requirements.txt
 ---> Running in 25b60ee404f6
Requirement already satisfied: numpy in /usr/local/lib/python3.6/site-packages (from -r requirements.txt (line 1)) (1.18.1)
Requirement already satisfied: pandas in /usr/local/lib/python3.6/site-packages (from -r requirements.txt (line 2)) (1.0.1)
Requirement already satisfied: scipy in /usr/local/lib/python3.6/site-packages (from -r requirements.txt (line 3)) (1.4.1)
Requirement already satisfied: netCDF4 in /usr/local/lib/python3.6/site-packages (from -r requirements.txt (line 4)) (1.5.3)
Requirement already satisfied: kafka in /usr/local/lib/python3.6/site-packages (from -r requirements.txt (line 5)) (1.3.5)
Requirement already satisfied: kafka-python in /usr/local/lib/python3.6/site-packages (from -r requirements.txt (line 6)) (2.0.1)
Requirement already satisfied: matplotlib in /usr/local/lib/python3.6/site-packages (from -r requirements.txt (line 7)) (3.1.3)
Requirement already satisfied: pytz>=2017.2 in /usr/local/lib/python3.6/site-packages (from pandas->-r requirements.txt (line 2)) (2019.3)
Requirement already satisfied: python-dateutil>=2.6.1 in /usr/local/lib/python3.6/site-packages (from pandas->-r requirements.txt (line 2)) (2.8.1)
Requirement already satisfied: cftime in /usr/local/lib/python3.6/site-packages (from netCDF4->-r requirements.txt (line 4)) (1.1.0)
Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in /usr/local/lib/python3.6/site-packages (from matplotlib->-r requirements.txt (line 7)) (2.4.6)
Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.6/site-packages (from matplotlib->-r requirements.txt (line 7)) (0.10.0)
Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.6/site-packages (from matplotlib->-r requirements.txt (line 7)) (1.1.0)
Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.6/site-packages (from python-dateutil>=2.6.1->pandas->-r requirements.txt (line 2)) (1.14.0)
Requirement already satisfied: setuptools in /usr/local/lib/python3.6/site-packages (from kiwisolver>=1.0.1->matplotlib->-r requirements.txt (line 7)) (39.2.0)
Removing intermediate container 25b60ee404f6
 ---> 8b52be2496e6
Step 7/10 : RUN python /pyart-master/setup.py build
 ---> Running in db303319adf2
python: can't open file '/pyart-master/setup.py': [Errno 2] No such file or directory
ERROR: Service 'postprocessing' failed to build: The command '/bin/sh -c python /pyart-master/setup.py build' returned a non-zero code: 2

On generating a plot , being redirected to login .

Docker:
https://www.docker.com/resources/what-container

Dockerize the data-retrieval service and have it correctly connect to kafka on kafka:9092.