cloudcheflabs / dataroaster Goto Github PK

License: Apache License 2.0

Shell 1.45% Java 96.53% Dockerfile 1.87% Python 0.10% Smarty 0.05%

dataroaster's Introduction

DataRoaster

DataRoaster is open source data platform running on Kubernetes.

DataRoaster Architecture

DataRoaster has a simple architecture. There are several operators in DataRoaster to install data platform components easily.

DataRoaster Operator handles REST requests to create custom resources.
After creating custom resources, DataRoaster Helm Operator and other operators like DataRoaster Trino Operator and DataRoaster Spark Operator will be notified and create data platform components.

Data Platform Components supported by DataRoaster

Components supported by DataRoaster:

Hive Metastore: standard data catalog in data lakehouses.
Spark Thrift Server: hive server compatible interface through which hive queries will be executed on spark.
Trino: popular query engine in data lakehouses.
Redash: popular BI tool.
Jupyterhub: multi-user version of jupyter notebook.
Kafka: popular event streaming platform.
Airflow: popular workflow.

Install DataRoaster

helm repo add dataroaster-operator https://cloudcheflabs.github.io/dataroaster-operator-helm-repo/
helm repo update;

helm install \
dataroaster-operator \
--create-namespace \
--namespace dataroaster-operator \
--version v3.0.8 \
--set image=cloudcheflabs/dataroaster-operator:4.3.0 \
--set dataroastermysql.storage.storageClass=oci \
dataroaster-operator/dataroaster-operator;

dataroastermysql.storage.storageClass is mysql storage class which needs to be replaced with one of your kubernetes cluster.

You need to copy the randomly generated admin password which will be used to access dataroaster api.

kubectl logs -f $(kubectl get pod -l app=dataroaster-operator -o jsonpath="{.items[0].metadata.name}" -n dataroaster-operator) -n dataroaster-operator | grep "randomly generated password for user";

Output looks like this.

...
randomly generated password for user 'admin': 9a87f65688a64e999e62c8c308509708
...

9a87f65688a64e999e62c8c308509708 is the temporary admin password which can be used for the first time and should be changed in the future.

To access DataRoaster from local, port-forward the service of dataroaster-operator-service.

kubectl port-forward svc/dataroaster-operator-service 8089 -n dataroaster-operator;

DataRoaster REST API

To access DataRoaster with REST, see DataRoaster REST API for more details.

DataRoaster Trino Controller

DataRoaster Trino Controller is used to control all the trino ecosystem components to build a trino gateway easily.

See DataRoaster Trino Controller for more details.

DataRoaster Trino Gateway

DataRoaster Trino Gateway is used to route the trino queries dynamically to downstream trino clusters.

See DataRoaster Trino Gateway for more details.

DataRoaster Trino Operator

DataRoaster Trino Operator is used to create/delete trino clusters easily.

See DataRoaster Trino Operator for more details.

DataRoaster Helm Operator

DataRoaster Helm Operator is used to install / upgrade / uninstall applications of helm charts easily.

See DataRoaster Helm Operator for more details.

DataRoaster Spark Operator

DataRoaster Spark Operator is used to submit and delete spark applications on kubernetes using custom resources easily. Not only spark batch job but also endless running applications like spark streaming applications can be deployed using dataroaster spark operator.

See DataRoaster Spark Operator for more details.

Community

DataRoaster Community Mailing Lists: https://groups.google.com/g/dataroaster

License

The use and distribution terms for this software are covered by the Apache 2.0 license.

dataroaster's People

Contributors

Stargazers

Watchers

Forkers

pongthep eric-doug unarmedcivilian phs2131 yuanyuan1und1 jmingc jacob119 mervynzhang nabaruns vcknorket vjain143 shyam-gole-clarista

dataroaster's Issues

where is spark-thrift-server-4.8.0-SNAPSHOT-spark-job.jar

hello，I want to try tun the spark demo，but cannot download spark-thrift-server-4.8.0-SNAPSHOT-spark-job.jar？ where is can be download ，thanks？

Add API to scale trino workers.

Update rest api provided by trino controller to install trino with different version.

To install trino with different version, trino docker image tag needs to be parameterized in rest api currently provided by trino controller.

Add DataRoaster Operator

DataRoaster Operator is used to controll all the data platform components running on kubernetes.
It consists of mysql server, helm operator and dataroaster server which provides rest api to the client.

Update prometheus jobs when trino configurations updated.

Everytime trino configuration updated, trino coordinator and workers in the dedicated trino cluster will be rollout restarted.
At that time, all the endpoints of trino coordinator and workers will get new addresses for their pods, so these changed endpoints addresses of trino coordinator and workers will be updated as prometheus jobs to prometheus configmap.

Add REST API to trino operator to get trino coordinator and workers mbean values like active executed query count, heap memory size.

Add REST API for dataroaster operator to create hive metastore automatically.

To create hive metastore, mysql server needs to be installed before installing hive metastore.
Add rest api for dataroaster operator to create mysql server and hive metastore automatically.

BUG: fix errros in trino gateway when python trino client connect to it.

Please share the link to download spark-thrift-server-4.8.0-SNAPSHOT-spark-job.jar

Please share the link to download spark-thrift-server-4.8.0-SNAPSHOT-spark-job.jar To Run Spark Application

Add custom resources of data platform components.

Custom resources of data platform components like hive metastore, redash, etc needs to be added.

Add Trino Gateway to proxy trino clusters.

Trino can be used for the cases of ETL, interactive(adhoc) and scheduled, etc.
We can create trino clusters for these usages individually. In order to load-balance trino clusters, and to support trino coordinator HA, trino gateway should be used. We can extend this concept to trino ecosystem.

Trino ecosystem consists of trino gateway, authenticator, admin and trino clusters, etc.
We can think of the following scenario which can happen in trino ecosystem.

clients(applications, redash, tableau, cli, etc) will request queries to trino gateway.
trino gateway will filter parameters of request to authenticate / authorize user with authenticator.
the queries requested will be routed dynamically to the downstream trino cluster for the individual usages of ETL, interactive, scheduled.
admin will create / update / delete trino clusters on kubernetes using trino operator.
admin will register / deregister / activate / deactivate trino cluster to trino gateway.

Add trino controller to control all the trino ecosystems.

Trino controller needs to control all the trino ecosystems.

create trino operator
create helm operator.
create nginx ingress controller.
create cert manager.
create trino gateway.
create trino cluster.
create prometheus.
create grafana.
scale out trino cluster.
register trino cluster to trino gateway.
deregister trino cluster from trino gateway.
detect exhausted trino cluster.

whereis the spark dockerfile

hello，where is the spark3 dockerfile，like cloudcheflabs/spark:v3.4.0 ，thanks！

Add API for redash, jupyterhub and kafka.

Get active executing queued query count from trino coordinator in every second and update it to redis cache.

Add Trino Operator

Trino operator will be used to create / update / delete trino clusters.
In addition to that, It should also monitor trino clusters and sometimes replace the exhausted trino workers with the new ones if detected.
We can also consider support of graceful shutdown when there are existing queries being executed which have to be executed before trino cluster shutdown.

Add Airflow Workflow

Airflow is one of the popular workflows for data lakehouse.
Airflow should support the following list:

dynamical dag workflow deployment using git-sync sidecar container.
support remote logging to s3.
KEDA based worker autoscaling.

Add REST API for dataroaster to create spark thrift server automatically.

To create spark thrift server, there are several things to be created beforehand.
dataroaster operator needs to create nfs server, rbac, s3 secret, and pvc automatically before spark thrift server created.

Add functionality to trino controller to detect exhausted trino clusters automatically.

Trino operator provides jmx apis to get a lot of mbean metrics exposed by trino coordinator and workers.
Probably, using this api, trino controller can detect exhausted trino cluster automatically.

Add specific apis for data platform components.

It is needed that specific apis for data platform components should be created to create these components automatically.
for instance,

/v1/spark_thrift_server for spark thrift server
/v1/trino for trino
/v1/hive_metastore for hive metastore
etc.

Monitor trino clusters in trino operator

Add monitoring of trino clusters in trino operator.
To monitor trino clusters in trino operator, the following functionalities can be added to trino operator(#1):

Bug: Watcher close error in trino operator.

if there is no action notified for a long time, watcher will be closed with the message like this.

com.cloudcheflabs.dataroaster.operators.trino.handler.TrinoClusterWatcher: close watcher

Add helm operator

Most of components provided by dataroaster is based on helm chart. With helm operator, these components can be installed on kubernetes easily.