PyTorch implementation of DeCUR.
| Pretrain dataset | Full model | Backbone only |
|---|---|---|
| SSL4EO-S12 | ResNet50-SAR/MS-ep100 | |
| GeoNRW | ResNet50-RGB/DEM-ep100 | |
| SUNRGBD | MiTB2-RGB/HHA-ep200 | MiTB2-RGB, MiTB2-HHA |
| SUNRGBD | MiTB5-RGB/HHA-ep200 | MiTB5-RGB, MiTB5-HHA |
Customize your multimodal dataset and your preferred model backbone in src/datasets/, src/models/ and src/pretrain_mm.py, and run
python pretrain_mm.py \
--dataset YOUR_DATASET \
--method PRETRAIN_METHOD \
--data1 /path/to/modality1 \
--data2 /path/to/modality2 \
--mode MODAL1 MODAL2
Apart from DeCUR, we also support multimodal pretraining with SimCLR, CLIP, BarlowTwins and VICReg.
If you are using distributed training with slurm, we provide some example job submission scripts in src/scripts/pretrain.
Multilabel scene classification with ResNet50 on BigEarthNet-MM:
$ cd src/transfer_classification_BE
$ python linear_BE.py --backbone resnet50 --mode s1 s2 --pretrained /path/to/pretrained_weights
Semantic segmentation with FCN on GeoNRW:
$ cd src/transfer_segmentation_GEONRW
$ python GeoNRW_MM_FCN_RN50.py --backbone resnet50 --mode RGB DSM mask --pretrained /path/to/pretrained_weights
Semantic segmentation with CMX on SUNRGBD and NYUDv2:
$ cd src/transfer_segmentation_SUNRGBD
$ python convert_weights.py # convert pretrained weights to CMX format
Then please refer to https://github.com/huaaaliu/RGBX_Semantic_Segmentation.
Simply load the pretrained weights from our pretrained models.
This project is licensed under the Apache 2.0 License - see the LICENSE file for details.
@article{wang2023decur,
title={DeCUR: decoupling common & unique representations for multimodal self-supervision},
author={Yi Wang and Conrad M Albrecht and Nassim Ait Ali Braham and Chenying Liu and Zhitong Xiong and Xiao Xiang Zhu},
journal={arXiv preprint arXiv:2309.05300},
year={2023}
}
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent