Features: Implement augmentation about cxrail-dev HOT 3 OPEN

So far, I think two strategies, auto augmentation & random augmentation, seem nice to try first

First of the two, auto augmentation looks nice but it is slightly different from what I expected

• It seems not a fully automated search. For example, PyTorch provides pre-trained auto augmentation
• Due to the drastic computation cost for optimizing augmentation, it assumes that a small portion of data can represent the whole data. This means that augmentation transfers from the subset to the whole

I referred

• https://arxiv.org/pdf/1805.09501.pdf
• https://pytorch.org/vision/main/generated/torchvision.transforms.AutoAugment.html
• https://towardsdatascience.com/augmentation-methods-using-albumentations-and-pytorch-35cd135382f8
• https://towardsdatascience.com/data-augmentations-in-torchvision-5d56d70c372e

For implementation, torchvision's AutoAugment or Albumentation's AutoAlbument look great to be to try first

from cxrail-dev.

After briefly searching the AutoAlbument,
(the AutoML tools for auto-augmentation of albumentations, https://albumentations.ai/docs/autoalbument/)
my mind became conservative about using the AutoAlbument

Before listing up the key points that changed my mind, it would be better to introduce the pros of AutoAlbument

• AutoAlbement implemented the Faster AutoAugment algorithm which is one of the popular and relatively new augmentation automation
• It has well organized documents (https://albumentations.ai/docs/autoalbument/how_to_use/), github repo (https://github.com/albumentations-team/autoalbument)
• The albumentations is a big, famous, and advanced project that is still improving

Despite these nice advantages above, there are two obstacles that made me conservative to use it

1. So far, AutoAlbument not supports multilabel problem (they support classification and segmentation
2. The AutoAlbument is implemented in pl-hydra format

Solving (1) might be possible by customizing the github code but it is hard to deny this is a time-consuming and difficult task
Thus, it should be very carefully compared between the difficulty of customizing AutoAlbument and its necessity for our purpose

(2) seems more complicated than (1) to me. Pl-hydra format makes AutoAlbument more user-friendly but since our code set also uses hydra, I can not guarantee it will make some conflict when we mix AutoAlbument in our code set
Thus, even if we divide searching best augmentation policy by the AutoAlbument & model training by our code set, the reliability of the suggestion that the found policy is not sub-optimal will rapidly disappear with a small difference between augmentation settings and model training settings
For example, the best augmentation policy by AutoAlbument for 100% CheXpert with resnet50 can not guarantee it is still best for 30% MIMIC with SWIN model

Additionally, searching best augmentation has an intrinsic problem in that it is a very cost-expensive task
Given that intrinsic problem, finding the best augmentation uses sampling, sub-policy strategy, and some assumptions

Thus, discuss best vs moderate augmentation in terms of the task's cost, difficulty, and necessity will be needed soon

from cxrail-dev.

I suggest the RandAugment instead of the Albumentations' AutoAlbument as the next job

In my point of view, RandAugment (RA) is quite simple but persuasive & the paper compares other popular automatic augmentation algorithms AutoAugment (AA), Fast AutoAugment (FAA), and Population-Based Augmentation (PBA) very well

Before diving into the RA, I referred to this paper (https://arxiv.org/pdf/1909.13719v2.pdf) and got a lot of insight from it
I recommend this paper not only to understand the RA but also to get insight into popular augmentation automatization strategies

The first key point : Simplicity

• RA is much more simple than AA, FAA, PBA
  • There are several differences between AA, FAA, PBA but they all searched a very large search space but RA has a much smaller search space
    
    (Table.1 from the RA paper)
  • The two parameters of RA are N (the number of transforms) and M (the magnitude for all the transformations)
• The RA search space also looks too simple to me (they even fixed all magnitude as a single value) but the scores are quite nice in the Table.1
• In the data field, the deeper domain knowledge always the better but what if a user lacks domain knowledge or is not familiar to customize augmentations? Simplicity could be very powerful in this case
• Even if the optimization method is highly efficient, a high-dimensional search space is quite a tough environment for finding the optimal choice
  • The author of the paper also emphasizes no necessity for a separate proxy task if a big advantage of the low-dimensional search space of RA
  • The author suggests naive grid-search is good enough to search the space

The second key point : Weak Assumption

(Figure.3 from the RA paper)

• The AA, FAA, PBA assume that proxy experiments (such as a sampled subset of data) are reflective of a bigger task
  • It is an intrinsic problem because searching for optimal augmentation with the whole dataset is computationally expensive
  • The difference between optimal distortion strength of small dataset and big dataset in Fig.3 well indicates the problem of strong assumption. More details are well described in the RA paper section 4.1
  • Especially, the paper mentioned that AA is not well functioning to the imagenet due to the proxy task's reflectiveness in section 4.4. It is important because our datasets are big enough to concern like imagenet and also, we use imagenet pre-trained model weights
• Reliability of proxy experiments is not only concerned with dataset size but also with model complexity
  • Fig.3 of the paper indicates optimal augmentation highly depends on the model size
  • Given that we have a plan to experiment with a wide variety of models, it could be wise not to assume a good augmentation for small-size models is good for big-size models too. In other words, the optimal choice for resnet50 could be sub-optimal for movilenet or efficientnet_B7 which means that one optimal augmentation needs for one model

from cxrail-dev.