berkeley-data / hpt Goto Github PK

dict(type='RandomGrayscale', p=0.2),

https://github.com/Berkeley-Data/OpenSelfSup/blob/taeil/configs/selfsup/moco/r50_v2_sen12ms_in_basetrain_20ep.py

The initial tricky part with using an RGB dataset (like RESISC) for evaluation is that the 3 channel input is not expecting RGB, but rather, the output of our input_modules (I think the right way to do this is to perform a bit of moco training on the target dataset where we freeze all params except an appropriate input_module -- but this will take a little engineering to set up).
So for now, it'll be easier if we use Sen1 and Sen2 for evaluation, and use their corresponding input_module that trained during moco pretraining.

Have a look at https://arxiv.org/pdf/2004.13390.pdf (section 4.1) which is based on sen12ms dataset.
"The optical and radar images were resampled to 10 m ground sampling distance and span 256×256 px in height and width. The original dataset uses tile-overlaps of 50%. For this work, we removed the overlap to ensure independence of support and query datasets, which yielded 200 306 128×128 px tiles."

They removed the overlap.

• Geography-Aware Self-supervised Learning Paper

• Functional Map of the World (FMoW)

• @TsungChinHanKen

• @SuryaTechie

• @taeil

• @oropezaev

https://git-scm.com/book/en/v2/Git-Tools-Submodules

This is created based on the following in Colorado's email:

Getting started goals:

• log onto llano
• with the HPT environment, complete the verify install section(https://github.com/cjrd/hpt#verify-install) of the README. You can skip the setup part of the README, I've already done that on llano for the crguest user.
• setup wandb (current system uses tensorboard, so it should be easy enough to use wandb...)

Just a thought: Since we are tracking the first part as a separate task, maybe we can perform the remaining two tasks together via zoom as it's a shared guest user account.

include the augmentation done by @surya.

dict(
# type='RandomAppliedTrans',
# transforms=[
# dict(
# type='ColorJitter',
# brightness=0.4,
# contrast=0.4,
# saturation=0.4,
# hue=0.4)
# ],
# p=0.8),

https://github.com/Berkeley-Data/OpenSelfSup/blob/taeil/configs/selfsup/moco/r50_v2_sen12ms_in_basetrain_20ep.py

• moco 1x1
• resnet50 with conv1x1
• resnet50
• moco (without 1x1)
• moco (with 1x1 but with random weight)

https://github.com/Berkeley-Data/hpt/blob/taeil/references/model_architectures.md

download and extract

Hey folks,

Let's take a look at a more straightforward contrastive learning baseline, using the following set of pretraining techniques, ie each of the following steps is a separate pretraining technique, ordered by complexity:

1. (pretraining) Have conv1 take 12 bands as input and use all 12 bands to do moco pretraining (this merges s1 and s2 data to be treated as a single "image"). Use whatever subset of the moco-v2 augmentations we have available across the 12 bands.
2. Do the same as (1.) except include cases where either s1 or s2 are not included for both the query and key view (e.g. a query might have s1+s2,s1, or s2 as input and the key would have the same input with a different set of augmentations). We can 0-pad the removed s1 or s2.
3. Do the same as (2.) except include cases where either s1 or s2 are not included for either the query and key view (e.g. a query might have s1+s2,s1, or s2 as input and the key might have s1+s2,s1, or s2 as input).

The idea here is that in (1.) we're doing instance discrimination where an input image is really a composition of 2 images. In (2.) we're doing instance discrimination where an input image can also be from only s1 or s2. In (3.), an input image can be from either (1.) or (2.).

• @SuryaTechie
• @TsungChinHanKen
• @oropezaev
• @taeil

• some brief tutorial in case...

Geography aware paper uses temporal images and they have some notations (just like many papers) specific to time as shown in the following figure:

Since our project uses spacially aligned (same location) images instead of temporal ones, it would be great if we have some notations which add value to the paper.

According to Colorado, most people will read the paper; they will glance at the first figure in the paper. We need to come up with some explainer diagrams and share with the group for feedback.

Based on the discussion in the team meeting (03/13 8:00 am), first we need to come up with a rough sketch of the current architecture. No need to focus on graphic creation.

I did a search for the issues in the wandb git repo. There were no such issues last year. Only this year, a couple of similar issues got reported. So, did testing by installing older versions starting from the latest version 0.10.22 . The issue is reproducible till 0.10.0. Below this version, the issue is not reproducible and is working fine with multiple workers.
Please install wandb version 0.9.7 which was released on Sep 8, 2020.
We can activate the conda env and install using 'pip install wandb==0.9.7'

Wandb team is releasing the patches frequently. They released 13 versions since Sep 8, 2020. Hopefully, they will fix the issue soon.

Originally posted by @SuryaTechie in #2 (comment)

• with subset data
• minimum transformation
• some hard-coded variables
• s1 and s2 as positive pair

,,,,

Check this dataset out. 180k triplets, georeferenced, multi band, multi modal, multi resolution: "SEN12MS -- A Curated Dataset of Georeferenced Multi-Spectral Sentinel-1/2 Imagery for Deep Learning and Data Fusion" https://arxiv.org/abs/1906.07789

download instructions

we should modify the main_train to convert and load in the same script to reduce the number of steps.

Getting the following warnings...Need to investigate and see if it's going to impact the results, if yes, we need to fix it.

/home/taeil/anaconda3/envs/hptest/lib/python3.7/site-packages/sklearn/metrics/_classification.py:1493: UndefinedMetricWarning: F-score is ill-defined and being set to 0.0 in labels with no true nor predicted samples. Use zero_division parameter to control this behavior.
average, "true nor predicted", 'F-score is', len(true_sum)
/home/taeil/anaconda3/envs/hptest/lib/python3.7/site-packages/sklearn/metrics/_classification.py:1493: UndefinedMetricWarning: F-score is ill-defined and being set to 0.0 in labels with no true nor predicted samples. Use zero_division parameter to control this behavior.
average, "true nor predicted", 'F-score is', len(true_sum)
/home/taeil/anaconda3/envs/hptest/lib/python3.7/site-packages/sklearn/metrics/_classification.py:1245: UndefinedMetricWarning: Precision is ill-defined and being set to 0.0 in labels with no predicted samples. Use zero_division parameter to control this behavior.
_warn_prf(average, modifier, msg_start, len(result))
/home/taeil/anaconda3/envs/hptest/lib/python3.7/site-packages/sklearn/metrics/_classification.py:1245: UndefinedMetricWarning: Recall is ill-defined and being set to 0.0 in labels with no true samples. Use zero_division parameter to control this behavior.
_warn_prf(average, modifier, msg_start, len(result))
Validation microPrec: 0.540000 microF1: 0.540000 sampleF1: 0.540000 microF2: 0.540000 sampleF2: 0.540000

updated data loader and script to run without an error but the result does not look correct.

The following code needs to be updated.

    for sample in loader:
        x = sample["image"]
        data = torch.stack([x.mean([1,2]), (x*x).mean([1,2])])

        results += data.sum(2)
        Nproc += data.shape[0]
        i += 1
        print("batch: {}/{}".format(i, NB))
        if i >= NB:
            break

We were using old version of torchvision.transform which only support PIL image.

https://pytorch.org/docs/1.6.0/torchvision/transforms.html#transforms-on-torch-tensor

either change to PIL image when loading the image or do our own transformation (which is probably bad idea).

Need to see if we can only upgrade torchvision only.

If try to transformToPIL, getting this error https://discuss.pytorch.org/t/error-while-using-transforms-topilimage-and-randomresizedcrop/12861

Colorado's change with custom data loader:
Berkeley-Data/OpenSelfSup#3

how to transfer 10 channel weights

• s2
• s1/s2
• s1

Sen12ms datasets evaluation (supervised models)

• document (single label). Make it more clear
• Document multi-label how the author did
• Document Sen12ms repo, 2 tasks (classification, semantic segmentation)
• Goal is to clearly explain to Colorado
• Simple vs full (Going over the code for dataset.py, simplified vs full (10 vs 17 class).
• Single vs multi
• S1, s2, rgb, vs s1/s2
• Different metrics
• K-mean usage (page 5)
• One class with no label (ice class)
• what does "probability" in this https://github.com/Berkeley-Data/SEN12MS/blob/taeil/splits/IGBP_probability_labels.pkl
  

the best model so far by @suryatechie

great job!

classification/main_train.py --exp_name finetune --data_dir data/sen12ms --label_split_dir splits --IGBP_simple --label_type multi_label --threshold 0.1 --model Moco --lr 0.00001 --decay 1e-5 --batch_size 64 --num_workers 4 --data_size 4096 --epochs 100 --pt_name vivid-resonance-73 --pt_type qe --pt_dir data/pretrained/moco --eval --use_s2 --use_s1

Create some initial augmentations like GaussianBlur, ElasticTransform, Blur, VerticalFlip, HorizontalFlip, RandomBrightnessContrast.

Write a simple util function to create positive pairs

The positive pair should be multiple images from different satellites at the same location.
The following is a positive pair:
ROIs1158_spring_lc_100_p101.tif
ROIs1158_spring_s1_100_p101.tif
ROIs1158_spring_s2_100_p101.tif

Note: The input can be a pkl file

Need code snippet and exploration around matching the dimensions.

• s1 and s2 to both 3 dimension
• s1 and s2 to some other dimension

Here is current code to load s1 image.

def load_s1(path, imgTransform):
    with rasterio.open(path) as data:
        s1 = data.read()
    s1 = s1.astype(np.float32)
    s1 = np.nan_to_num(s1)
    s1 = np.clip(s1, -25, 0)
    if not imgTransform:
        s1 /= 25
        s1 += 1
    s1 = s1.astype(np.float32)
    return s1

Please help modify

• each user should be able to use their own apikey
• track loss or summary

This issue was mentioned in our meeting with Colorado last Friday. Pixel level training refers to using Land Cover Images as labels for each pixel. There are 4 images for land cover (4x256,256). The first slice correspond to the Full Label classification. The other 3 assign other values to some of the classes that we know.

# dict(type='RandomResizedCrop', size=224, scale=(0.2, 1.)),

https://github.com/Berkeley-Data/OpenSelfSup/blob/taeil/configs/selfsup/moco/r50_v2_sen12ms_in_basetrain_20ep.py

Check this dataset out. 180k triplets, georeferenced, multi band, multi modal, multi resolution: "SEN12MS -- A Curated Dataset of Georeferenced Multi-Spectral Sentinel-1/2 Imagery for Deep Learning and Data Fusion" https://arxiv.org/abs/1906.07789

download instructions

using 1 or 2, pre-training collects loss. With 4, it failed to collect loss (or failed to calculate loss). Need to debug and see what is happening.

repo
wandb

pretrained models are downloaded and placed on s3 (s3://sen12ms/pretrained_sup) for faster download along with other pre-trained models.

1. use the pretrained models
2. run various prediction and explore different metrics for baselines

Once this is done, please convert this issue to the discussion for documentation purpose.

no epoch progress for pre-training (sen12ms)

currently it saves all the checkpoints which use up project storage on wandb too fast. need to only save the best model to the wandb folder.

assuming our method will work, we need to start writing a paper and share the draft / structure with Colorado and professors.

Here is current draft.

Yeah - Sen12ms may not be the perfect evaluation because of the mislableling issue, but it's a good diagnostic to make sure our system is working properly -- and we can report the results either way.

it'll be easier if we use Sen1 and Sen2 for evaluation, and use their corresponding input_module that trained during moco pretraining.

A 200 epoch model is at:
/scratch/crguest/vivid-resonance-73_sen12ms_no_aug_200epoch.pth
where the wandb training is here: https://wandb.ai/cjrd/BDOpenSelfSup-tools/runs/3qjvxo2p?workspace=user-cjrd

(transfer learning)
run sen12ms training on s1 and s2 two separately using same pre-trainining model

1. fine tune using small subset of training data to measure.
2. prepare small subset of training data set
3. same validation set
4. scene validation with top1 and top5 (compare to baseline)

Additional Update on March 26th

• resnet50+1x1 with transferred weight for input module

pretrained models are downloaded and placed on s3 (s3://sen12ms/pretrained_sup) for faster download along with other pre-trained models.

1. use the pretrained models
2. run various prediction and explore different metrics for baselines

Once this is done, please convert this issue to the discussion for documentation purpose.

Berkeley-Data/OpenSelfSup#3

Evaluating a subset of Test data and reducing the 128 output component to 2 or 3 component to plot them and describe the visual classification

I think everyone should replace their p3 instance by this one since we may not use the previous one any more.

• @SuryaTechie
• @TsungChinHanKen
• @oropezaev

updated the instructions here.

currently running on small dataset. Need to expand the training on bigger dataset.

# dict(type='RandomHorizontalFlip'),

https://github.com/Berkeley-Data/OpenSelfSup/blob/taeil/configs/selfsup/moco/r50_v2_sen12ms_in_basetrain_20ep.py

• saving plot images to wandb.run.dir should be simple enough as well.

according to Colorado, full train set includes test set.
Please help fix train set so it doesn't include test set.

Also, test set contains only 8 classes. Please help confirm and how to address this issue.