dennishylau / 5003-project Goto Github PK

View Code? Open in Web Editor NEW

2.0 2.0 1.0 1.93 MB

HKUST MSBD-5003 Group Project

Shell 35.98% Dockerfile 2.52% Python 21.56% Jupyter Notebook 36.25% HCL 3.69%

5003-project's People

Contributors

Stargazers

Watchers

Forkers

thomas-tf

5003-project's Issues

deploy: explore azure

bug: pyspark worker uses another python version

feat: Enable Grafana plug-ins

Enable Grafana plug-ins

For unsigned plug-ins, need to amend the configuration for allowing the unsigned plug-ins to load
For core plug-ins that built into grafana, user currently not allowed to install, need to install the plug-ins in Dockerfiles

feat: Kaggle dataset schema

feat: db schema setup

UX: Grafana wireframe

feat: stream generator

feat: data ingestion program

reads csv and write records to kafka base on the record timestamp. Support a parameter as input that controls the speed up of writing data to kafka.

feat: Prelimi Plot

feat: Grafana layout

fix: Amend grafana datasources configuration

Change host from http://localhost:5432 to db-timescale:5432
Disable the SSL mode

feat: Kafka ingestion

fix: timescale db extension not enabled by default

research: visualization methods (with screen cap)

feat: spark pipeline with live inference

Build data pipeline to consume kafka and make inference using NLTK/TextBlob on polarity. Write results to TimescaleDB, schema to be aligned with team members.

hotfix: zookeeper healthcheck command

kafka broker depends on the health condition of zookeeper, but zookeeper's healthcheck command is not working, causing the broker to never start.