This project is a PoC/example to show automation of large-scale Cloud Infra optimization with the orchestration tech and BO to decouple cross-domain expertise and accelerate the experiments to a new level.

Below is the arch digram:

Cloud-pipeline(README) automated expensive experiments by wiring the circle:

• With given parameter as input, Perform Parameter Applying towards the Cloud Infra System
• Running Backgroud Traffic load on the system with predefined Applications: code
• Instantiating a given Application under scheduling policy defined by the parameter and Run a Benchmark defined here
• Collect Benchmark data and cleanup env to be ready for next epoch

With the circle connected by Cloud-pipeline, we could run optimization with Bayesian Optimization in a jupyter notebook: demo

Those things put as predefined ones while should be inputs of the toolchain itself:

• The given parameter to tune
• The background traffic
• The optimization target for application workload & its Benchmark(desc here)

Also, a dashboard(code: frontend, backend) to help visualization of the training process and the outcome of the tuning was created below is a screen record for it: https://vimeo.com/497995660

And, not just calling them in python, a CLI for cloud-pipeline was also created to easily debug, operate the experiment in a handy way, here is a screen record for that: https://vimeo.com/497997340