Code Monkey home page Code Monkey logo

scp756-221 / term-project-data-miners Goto Github PK

View Code? Open in Web Editor NEW
1.0 2.0 2.0 487 KB

Implemented User-Music-Bookstore microservice architecture which is deployed on AWS Elastics Kubernetes Service where several REST APIs are used to fetch and store the data to and from AWS DynamoDB. Used Gatling to do load testing on the system and viewed the graphical metrics on Grafana

Makefile 27.12% Shell 17.34% Dockerfile 2.58% Python 44.18% Scala 3.16% Batchfile 5.61%
aws-dynamodb aws-eks-cluster gatling-simulations grafana-dashboard kiali microservices-architecture rest-api

term-project-data-miners's Introduction

Dataminers: CMPT 756 Main Project Directory

This is the course repo for CMPT 756 (Spring 2022)

You will find resources for your assignments and term project here.

1. Prerequisites

Note: This project has been developed on the local machine and not in the Ubuntu Container(./tools/shell.sh).

  • Install istioctl, kubectl, eksctl, helm on your local machine.
  • Create a personal access token (PAT) for your GitHub account. You will need the three scopes: [read:packages], [write:packages] and [delete:packages]. Save your token in the file cluster/ghcr.io-token.txt
  • Instantiate configuration template:
    • Make a copy of tpl-vars-blank.txt named tpl-vars.txt, in the same directory.
    • On the line starting ZZ-REG-ID=, append your GitHub userid.(Note that there are no spaces around the = sign.)
    • Similarly, fill ZZ-AWS-REGION=us-west-2, ZZ-AWS-ACCESS-KEY-ID=<access-key-of-your-acc>, ZZ-AWS-SECRET-ACCESS-KEY=<secete-key-of-your-acc>
  • Fill the REGID= with your GitHub userid in dataminers.mak

2. Getting Started

Instantiate the template files.

$ make -f dataminers.mak initrepo

Starting the EKS Cluster.

This step can take some time

$ make -f dataminers.mak starteks

Create namespace inside each cluster and set each context

$ make -f dataminers.mak initns

To view the current context and all AWS clusters and nodegroups

$ make -f dataminers.mak listeks

Creating the DynamoDB Tables

$ make -f dataminers.mak initdb

Deploy User, Music and Bookstore service

$ make -f dataminers.mak deploy

Start simulation with desired number of Users

$ make -f dataminers.mak startsimulation NUM_OF_USERS=5

View the simulation with Grafana, Prometheus, Kiali

$ make -f dataminers.mak simulation-url

Stop Simulations

List the Simulation Docker Container

$ docker ps -a

Stop the containers running with ghcr.io/scp-2021-jan-cmpt-756/gatling:3.4.2 Image

$ docker stop <container-id>

term-project-data-miners's People

Contributors

dhruv007patel avatar github-classroom[bot] avatar himalya avatar kishan-thumar avatar rahilbalar98 avatar

Stargazers

 avatar

Watchers

 avatar  avatar

term-project-data-miners's Issues

Setup

Group formation on Coursys, Github, and creation of Kanban project board for tracking daily status.

Adding Methods to Database Service

Sub Task to the Main Issue #5

Adding the method to read all records, and write all records ( via .csv file ) in the Dynamo DB. This particular step will involve the following task.

  • Configuring the table in the cloudformation file to create a table on the DynamoDB.
  • Creating a method to read all the records present in the Database.
  • Creating a method to write all records written in a .csv file into the Database.

Deploying the services in the Kubernetes.

To deploy the DB, S1, S2, Bookstore services on the Kubernetes, in order to achieve that I and @dhruv007patel will be looking into the following sub-task.

  • Setting up the minikube cluster & cloud cluster
  • Installing Service mesh
  • Tunneling into your cluster in the minikube & cloud
  • Deploying the services.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.