Possible Options:

• Flask
• FastAPI
• MLServer

Currently leaning towards MLServer or FastAPI.

Important:
Decide on why this Framework was chosen.

The server should use the newest/best version of the model based on arbitrary metric - assigned through weights and biases.
The function should use the model to make a prediction.
This prediction (image) is then returned and given to the Streamlit UI

• Depends on #5,#6, #15

Find out, how the GCS can be used in lightningAI, a possible approach could include:
https://github.com/Lightning-AI/litdata,
sadly https://lightning.ai/docs/app/stable/workflows/mount_cloud_object_store.html states, that google cloud buckets aren't supported now.

• Define the Hyperparameters that influence the model performance. (i.e. Learning Rate, Batch Size, etc.)
• Track the Hyperparameters per Run in W&B.

We could theoretically calculate the free memory and adjust the batch size accordingly to save money.

• config file
• define sweeps settings

The Artifact Registry is currently on europe-west9 (France while the Cloud Build runs on europe-west1 (Belgium).
Since our docker images are fairly large, transferring them between datacenters is probably expensive so we should have them in one location.

In order to train a diffusion model we need a dataset.
The data can either be static or continously added.

• Select Dataset and give reasons for why we use it
• Upload dataset to GC
• #4

Hollywood & Vice keep down the price!

Options:

• GUnicorn
• Guvicorn
• uvicorn
• Waitress
• uWSGI

Note: uWSGI is only to be used when you know why you're using it. I most likely don't.

Possibilities:

• Get model from weights and biases, and get model directly. The Tag is deployment
• This happens over GitHub Actions.

• When #5 is done - create an example Server for the Deployment.

• Depends on #4
• Depends on #7
• Depends on #5

How does the pathing work? significantly different from normal machine?

Create a new account for our team and invite all members.

Name ist programm

Use a timestamp instead of 0-4.jpg

Should be:

• Losses
  - evaluation
• Human Feedback
  - for example, 5 stars to every picture and evaluate the result

Weights & Biases:

Positive:

• Sweeps (with the config files, where we have all dependencies)
• The experiments can be well structured and very well followed
• Integration into existing frameworks
• Good and professional visualizations
• supports team collaboration
• Model Saving and Reproducibility are possible
• Scalability: It supports scaling from local experiments to large-scale deployments smoothly, making it suitable for both research and production environments.
• System performance could be checked
• [ ]

Negative:

• Costs
• Dependency and Privacy

Comet.ml

• Comprehensive Experiment Management: Tracks code, experiments, and results
• Integration: Supports multiple frameworks and languages, facilitating integration into existing projects without major adjustments.
• Community and Support: Provides good documentation and has a supportive community, which can be helpful for troubleshooting and best practices.

PyTorch

HYDRA

Pros

• Simplification of Configuration: HYDRA allows you to create hierarchical configurations dynamically by composing and overriding them from the command line. This simplifies managing configurations for complex applications requiring various setups for different environments or tasks.
• Flexibility: It supports a wide range of data sources for configuration files, including YAML, and it can seamlessly merge configurations from multiple sources. This flexibility is particularly useful in research and development environments where rapid testing of configurations is standard.
• Plugin System: HYDRA includes a robust plugin system, allowing seamless functionality extension and integration with other tools and frameworks.
• Command Line Integration: Configurations can be easily adjusted or overridden from the command line, making it convenient to quickly change parameters without altering code or configuration files directly.

Cons

• Learning Curve: Although HYDRA is very powerful, it has a learning curve. Users must become familiar with its configuration composition and overriding principles to use it effectively.

• Integration Effort: While integrating HYDRA into existing projects offers many long-term benefits, the initial setup and integration might require significant effort, especially in projects not designed to use external configuration management.

• Dependency and Complexity: Adding HYDRA introduces another dependency into the project, which can increase the complexity and potentially impact the build and deployment processes.

• Overhead for Smaller Projects: For smaller projects or those that do not require frequent configuration changes or extensive experimentation, the overhead of implementing and maintaining a HYDRA-based configuration might not be justified.

• PyTorch - framework supports techniques used in continual learning, like experience replay, elastic weight consolidation (EWC), and other regularization methods

• Avdvertorch

Alibi Detect

Pros

• Versatility: Alibi Detect provides a wide range of detection algorithms suitable for various data types, including tabular data, text, images, and time series.
• Framework Compatibility: It supports PyTorch backends for drift detection, making it flexible for use in different machine learning stacks.
• Online and Offline Detection: The library supports both online and offline detection methods, which means it can handle streaming and batch data.
• Good documentation
• Ease of Integration: It can be easily integrated into existing data processing pipelines and machine learning workflows.

Cons

• Complexity: The wide range of features and algorithms might be overwhelming for beginners or practitioners new to drift, outlier, and adversarial detection.
• Scalability: Depending on the models' complexity and the data's size, scaling the detectors for high-throughput systems might require additional engineering.
• Resource Intensive: Some detection algorithms, especially for high-dimensional data such as images and time series, can be resource-intensive, requiring significant computational power.
• [ ]

The model registry will allow us to fetch previously trained models and compare them.
We can also use it to provide the model service with our currently best model.

We could also only use GC or another provider, but since we are using W&B for the metrics, it makes a lot of sense to also store the models there.

• Save them to the W&B registry from #9

To be served on Google Cloud container Registry

• Depends on #11 and #9 .

• Dataset class #1
• Model class
• Pytorch Lightning Trainer
• #12

• Run GitHub Action when the tag run_training is pushed
• We are running out of space with the Pytorch image, so let's switch to a Nvidia / CUDA based one
• Still running out of space, so perhaps switch back to the normal docker build command?
• Implement Google Cloud Build pipeline
• Create secrets for W&B and Cloud Bucket keys
• Start Lightning AI studio with GPU and run training
• Track metrics to W&B project #29
• #12

Currently the CI pipeline needs to be updated manually with a docker build command.

docker build -t europe-west9-docker.pkg.dev/algorithmic-quartet/training-pipelines/lightning-executor:latest
docker push europe-west9-docker.pkg.dev/algorithmic-quartet/training-pipelines/lightning-executor:latest

We could also automate this by building the container whenever something in the CI folder has changed.

• Make a prediction happen
• Get the prediction from the Google cloud storage

• Upload dataset to a storage according to: https://cloud.google.com/storage/docs/uploading-objects?hl=de

Why we decided to choose GC:

Positive:

• data availabilty / distributed, collaboration is a breeze
• variety of storage modes -> for every need / price
• Credits granted for GC

Negative:

• dependence of GC

The Hows and whys will be explained further by Gerome

• Depends on #7

• look into actual hugging face models, resp. into stable diffusion: https://huggingface.co/docs/diffusers/main/en/tutorials/basic_training#train-a-diffusion-model or https://www.kaggle.com/code/digvijayyadav/pokemon-generator-gans

It's defintily going to be a GAN!

• Define a trigger from the W&B Model Registry (#9) (Refer to https://docs.wandb.ai/guides/model_registry)
• Upon trigger, have a Google Cloud Run Container compute the best model
• Remove deployment alias from all models
• Define a "deployment" alias for the new best model

The Docker image takes a long time to build (>15 minutes). A large portion of this comes down to the python libraries, especially the pytorch dependencies!

We need to find some way to reduce this to a reasonable duration.

• Depends on / Related to #26
• Create a base-image which is only updated if the requirements or Dockerfile has changed (See this Stackoverflow Post
  Then create a training image which runs on top of this base image which is re-built everytime.
• Find a way to cache the installs between runs.
• Look into poetry

Prepare the Developer Environment for the Frontend.

• create Project in W&B
• define run-names for experiments
• settings for W&B in code