Optimize the dataloading pipeline for speed and check for issues. Based on the script that @julien Siems wrote can you please try if it is possible to get a Dataloader object without any memory issues on PyTorch from the tf_records iterator during training (https://github.com/automl/AutoDLComp19/blob/master/src/image/before_code_structure/model.py).

Add tensorboard to the model to visualize image model's input

Check online_meta.py if there are any logical errors

Investigate/optimize the general scheduling/stopping criteria of model.py

Add cluster compatibility for non-submission experiments, i.e. add $SLURM_ARRAY_JOB_ID to the folder name where logs are written to avoid overwriting the results.

Run fine-tuning experiments on these models from #5

Add the really small datasets from the competition starting kit to the repository.

Review if the dataloading we discussed still has issues and write a message to the competition organizers

Train shake-shake models on PyTorch 1.0.1, save them and add the scripts for loading the checkpoints to the codebase.

Run experiments on which channel to exclude for the 4-channel dataset.
Arber: If this dataset is CMYK, we can map the (None, None, 4) tensor to RGB (None, None, 3) as pointed out here https://stackoverflow.com/questions/14088375/how-can-i-convert-rgb-to-cmyk-and-vice-versa-in-python

@anupam Kakkar, @debayan Sen @manav Madan: If the data of this dataset is not normalized and the pixel range is (0, 100) it is indeed CMYK.

Research and review more sophisticated online-optimization strategies (weight regularization, optimizers, lr-schedule, ...)

Make an interactive plot to quickly looup the minutes or score percentage given score percentage or minutes + offset.
The input should be:
input type - percentage/minutes
t0 - The not counted seconds in the beginning
tmax - Total amount of time available
t1 - Lower bound of the region to convert
t2 - Upper bound of the region to convert

This should return the timer/percentage

• Find a way to include this as interactive graph into the Readme.md
• Implement it

Add the option to use mixed precision training to the pipeline.

• Add the nvidia apex lib to the environment.yaml for automatic install
• Add the necessary files to the config.hjson the library needs to function
• Add an option to the config.hjson to use/not use mixed precision training

• Check capabilities
• Tick off 1

Run experiments on finetuning strategies (including exploring the hyperparameters) [also specific to test dataset]

Create a preprocessing and augmentation pipeline

As discussed yesterday, it would be great if we have a zoo/portfolio of pretrained models with different hyperparameters. I think considering the time left we need something that trains relatively quick. You can use WideResNet since it is a shallow model and trains for less than 3h achieving a test error of around 3% on CIFAR-10. Another reason why I chose this model is because it is trained with a step leaning rate schedule. Of course, cosine annealing might be a better choice for training a network but if we want to fine-tune from these pretrained models and keep a decent accuracy in the very first beginning we need to keep in mind that increasing the learning rate after it has decayed gets you away from that good local minima and the accuracy drops. On the other hand, if the pretrained models had a leaning rate different from 0 at the end of their training we can directly use that value when starting to fine-tune. Lets keep it like this for now, later we can experiment with cosine annealing with lr_min != 0 at the end of training.
At this point I would suggest to start running BOHB with min_budget=max_budget=200 epochs and 100 iterations. Optimize lr, weight decay and mixup interpolation factor (I will send you the code, so you can find them in the arguments. Then you just integrate this search space with the BOHB scripts you have already been running for DARTS.). The ranges can be [0.01, 1] (log scale), [1e-5, 1e-3] (log scale), [0, 1], respectively. Please make sure to save the model parameters and optimizer state after training has finished for each sampled configuration. In the end we will have 100 pretrained models on CIFAR10 that we can use later.

Identify which datasets transfer well to each other.

Implement random initialization (and not loading from pretrained). We need this to determine online at which point we want to evaluate our model.

Since black had issues, include a different autoformater and also include isort again

Review and reasearch how we should handle unbalanced data

ResNet-101, ResNet-152, DenseNet-161, ResNeXt-50-32x4d, ResNeXt-101-32x8d from torchvision, to the codebase so they can be easily called and fine-tuned.