cdtrans / cdtrans Goto Github PK

View Code? Open in Web Editor NEW

311.0 311.0 40.0 10.44 MB

[ICLR2022] CDTrans: Cross-domain Transformer for Unsupervised Domain Adaptation

License: MIT License

Python 92.40% Shell 7.60%

cdtrans's Introduction

CDTrans: Cross-domain Transformer for Unsupervised Domain Adaptation

Introduction

This is the official code of CDTrans: Cross-domain Transformer for Unsupervised Domain Adaptation.

Results

Table 1 [UDA results on Office-31]

Methods	Avg.	A->D	A->W	D->A	D->W	W->A	W->D
Baseline(DeiT-S)	86.7	87.6	86.9	74.9	97.7	73.5	99.6
Baseline(DeiT-S)	86.7	model		model		model
CDTrans(DeiT-S)	90.4	94.6	93.5	78.4	98.2	78	99.6
CDTrans(DeiT-S)	90.4	model	model	model	model	model	model
Baseline(DeiT-B)	88.8	90.8	90.4	76.8	98.2	76.4	100
Baseline(DeiT-B)	88.8	model		model		model
CDTrans(DeiT-B)	92.6	97	96.7	81.1	99	81.9	100
CDTrans(DeiT-B)	92.6	model	model	model	model	model	model

Table 2 [UDA results on Office-Home]

Methods	Avg.	Ar->Cl	Ar->Pr	Ar->Re	Cl->Ar	Cl->Pr	Cl->Re	Pr->Ar	Pr->Cl	Pr->Re	Re->Ar	Re->Cl	Re->Pr
Baseline(DeiT-S)	69.8	55.6	73	79.4	70.6	72.9	76.3	67.5	51	81	74.5	53.2	82.7
Baseline(DeiT-S)	69.8	model			model			model			model
CDTrans(DeiT-S)	74.7	60.6	79.5	82.4	75.6	81.0	82.3	72.5	56.7	84.4	77.0	59.1	85.5
CDTrans(DeiT-S)	74.7	model	model	model	model	model	model	model	model	model	model	model	model
Baseline(DeiT-B)	74.8	61.8	79.5	84.3	75.4	78.8	81.2	72.8	55.7	84.4	78.3	59.3	86
Baseline(DeiT-B)	74.8	model			model			model			model
CDTrans(DeiT-B)	80.5	68.8	85	86.9	81.5	87.1	87.3	79.6	63.3	88.2	82	66	90.6
CDTrans(DeiT-B)	80.5	model	model	model	model	model	model	model	model	model	model	model	model

Table 3 [UDA results on VisDA-2017]

Methods	Per-class	plane	bcycl	bus	car	horse	knife	mcycl	person	plant	sktbrd	train	truck
Baseline(DeiT-B)	67.3 (model)	98.1	48.1	84.6	65.2	76.3	59.4	94.5	11.8	89.5	52.2	94.5	34.1
CDTrans(DeiT-B)	88.4 (model)	97.7	86.39	86.87	83.33	97.76	97.16	95.93	84.08	97.93	83.47	94.59	55.3

Table 4 [UDA results on DomainNet]

Base-S	clp	info	pnt	qdr	rel	skt	Avg.	CDTrans-S	clp	info	pnt	qdr	rel	skt	Avg.
clp	-	21.2	44.2	15.3	59.9	46.0	37.3	clp	-	25.3	52.5	23.2	68.3	53.2	44.5
clp	model						37.3	clp	model	model	model	model	model	model	44.5
info	36.8	-	39.4	5.4	52.1	32.6	33.3	info	47.6	-	48.3	9.9	62.8	41.1	41.9
info	model						33.3	info	model	model	model	model	model	model	41.9
pnt	47.1	21.7	-	5.7	60.2	39.9	34.9	pnt	55.4	24.5	-	11.7	67.4	48.0	41.4
pnt	model						34.9	pnt	model	model	model	model	model	model	41.4
qdr	25.0	3.3	10.4	-	18.8	14.0	14.3	qdr	36.6	5.3	19.3	-	33.8	22.7	23.5
qdr	model						14.3	qdr	model	model	model	model	model	model	23.5
rel	54.8	23.9	52.6	7.4	-	40.1	35.8	rel	61.5	28.1	56.8	12.8	-	47.2	41.3
rel	model						35.8	rel	model	model	model	model	model	model	41.3
skt	55.6	18.6	42.7	14.9	55.7	-	37.5	skt	64.3	26.1	53.2	23.9	66.2	-	46.7
skt	model						37.5	skt	model	model	model	model	model	model	46.7
Avg.	43.9	17.7	37.9	9.7	49.3	34.5	32.2	Avg.	53.08	21.86	46.02	16.3	59.7	42.44	39.9

Base-B	clp	info	pnt	qdr	rel	skt	Avg.	CDTrans-B	clp	info	pnt	qdr	rel	skt	Avg.
clp	-	24.2	48.9	15.5	63.9	50.7	40.6	clp	-	29.4	57.2	26.0	72.6	58.1	48.7
clp	model						40.6	clp	model	model	model	model	model	model	48.7
info	43.5	-	44.9	6.5	58.8	37.6	38.3	info	57.0	-	54.4	12.8	69.5	48.4	48.4
info	model						38.3	info	model	model	model	model	model	model	48.4
pnt	52.8	23.3	-	6.6	64.6	44.5	38.4	pnt	62.9	27.4	-	15.8	72.1	53.9	46.4
pnt	model						38.4	pnt	model	model	model	model	model	model	46.4
qdr	31.8	6.1	15.6	-	23.4	18.9	19.2	qdr	44.6	8.9	29.0	-	42.6	28.5	30.7
qdr	model						19.2	qdr	model	model	model	model	model	model	30.7
rel	58.9	26.3	56.7	9.1	-	45.0	39.2	rel	66.2	31.0	61.5	16.2	-	52.9	45.6
rel	model						39.2	rel	model	model	model	model	model	model	45.6
skt	60.0	21.1	48.4	16.6	61.7	-	41.6	skt	69.0	29.6	59.0	27.2	72.5	-	51.5
skt	model						41.6	skt	model	model	model	model	model	model	51.5
Avg.	49.4	20.2	42.9	10.9	54.5	39.3	36.2	Avg.	59.9	25.3	52.2	19.6	65.9	48.4	45.2

Requirements

Installation

pip install -r requirements.txt
(Python version is the 3.7 and the GPU is the V100 with cuda 10.1, cudatoolkit 10.1)

Prepare Datasets

Download the UDA datasets Office-31, Office-Home, VisDA-2017, DomainNet

Then unzip them and rename them under the directory like follow: (Note that each dataset folder needs to make sure that it contains the txt file that contain the path and lable of the picture, which is already in data/the_dataset of this project. The 'Real World' domain directory name of the Office-Home should be renamed to 'Real_World' for dataset loading. Otherwise you may encounter "FileNotFoundError: [Errno 2] No such file or directory" )

data
├── OfficeHomeDataset
│   │── class_name
│   │   └── images
│   └── *.txt
├── domainnet
│   │── class_name
│   │   └── images
│   └── *.txt
├── office31
│   │── class_name
│   │   └── images
│   └── *.txt
├── visda
│   │── train
│   │   │── class_name
│   │   │   └── images
│   │   └── *.txt 
│   └── validation
│       │── class_name
│       │   └── images
│       └── *.txt

Prepare DeiT-trained Models

For fair comparison in the pre-training data set, we use the DeiT parameter init our model based on ViT. You need to download the ImageNet pretrained transformer model : DeiT-Small, DeiT-Base and move them to the ./data/pretrainModel directory.

Training

We utilize 1 GPU for pre-training and 2 GPUs for UDA, each with 16G of memory.

Scripts.

Command input paradigm

bash scripts/[pretrain/uda]/[office31/officehome/visda/domainnet]/run_*.sh [deit_base/deit_small]

For example

DeiT-Base scripts

# Office-31     Source: Amazon   ->  Target: Dslr, Webcam
bash scripts/pretrain/office31/run_office_amazon.sh deit_base
bash scripts/uda/office31/run_office_amazon.sh deit_base

#Office-Home    Source: Art      ->  Target: Clipart, Product, Real_World
bash scripts/pretrain/officehome/run_officehome_Ar.sh deit_base
bash scripts/uda/officehome/run_officehome_Ar.sh deit_base

# VisDA-2017    Source: train    ->  Target: validation
bash scripts/pretrain/visda/run_visda.sh deit_base
bash scripts/uda/visda/run_visda.sh deit_base

# DomainNet     Source: Clipart  ->  Target: painting, quickdraw, real, sketch, infograph
bash scripts/pretrain/domainnet/run_domainnet_clp.sh deit_base
bash scripts/uda/domainnet/run_domainnet_clp.sh deit_base

DeiT-Small scripts Replace deit_base with deit_small to run DeiT-Small results. An example of training on office-31 is as follows:

# Office-31     Source: Amazon   ->  Target: Dslr, Webcam
bash scripts/pretrain/office31/run_office_amazon.sh deit_small
bash scripts/uda/office31/run_office_amazon.sh deit_small

Evaluation

# For example VisDA-2017
python test.py --config_file 'configs/uda.yml' MODEL.DEVICE_ID "('0')" TEST.WEIGHT "('../logs/uda/vit_base/visda/transformer_best_model.pth')" DATASETS.NAMES 'VisDA' DATASETS.NAMES2 'VisDA' OUTPUT_DIR '../logs/uda/vit_base/visda/' DATASETS.ROOT_TRAIN_DIR './data/visda/train/train_image_list.txt' DATASETS.ROOT_TRAIN_DIR2 './data/visda/train/train_image_list.txt' DATASETS.ROOT_TEST_DIR './data/visda/validation/valid_image_list.txt'

Acknowledgement

Codebase from TransReID

cdtrans's People

Contributors

Stargazers

Watchers

Forkers

jing--li liuyangriverchen slpal dear-john miss-rain cwhgn zhaoxin94 atlasgooo2 profbressan jin-huihuang xiu10086 createrll jasonzhan715 scottjingtt csmlzzy linkserendipity suvarnak nickniulang yuhao20 xzm2002 bigcowpeking jordanresearch agbeli alphapav jiang20 junchengberry wgqtmac wikiloa emerld2011 blueacacia kangtianxing fazlicodes mamengyao1999 acupofair yuntaodu cdgyp wkq-wukaiqi siaimes minjepark2 xkxrobot

cdtrans's Issues

Questions about reproducing on VisDa

Have some questions about running the model on VisDa Dataset.
Firstly when we click on the provided link,should we also download the tar files in openset directory also.

Also,based on the train.txt and val.txt present in data folder,it seems that the openset classification wasn't addresed by the project. if that's the case did you try to solve the openset classification task.

Finally, in order to reproduce this on custom dataset,what steps are necessary to reproduce ? Can the code be executed on any pretrained model or is it necessary to use transformers? Plus can the project be used for both commerical as well as non commmerical purposes?

Error when training on DomainNet

Hello!
I have an issue when I trains a DeiT-S with CDTrans on DomainNet.
An error was occured, and I analyzed the error by making use of breakpoint().
Based on the result of debugging, I could see that the image pairs don't remain after image filtering process.
In paper for CDTrans, the results on DomainNet can be seen, and I think training should also be possible on this dataset.

My question is what should I do for training on DomainNet without error.
Should I modify the parameter 'topk'? or Should I modify WITH_PSEUDO_LABEL_FILTER in uda.yml?
If not, is there any other way to handle this problem?

Thank you for reading it!

Does the 'Baseline' in your paper mean "Deit-Base"?

Hi, I appreciated it for your great work! :)

I have a question. Does 'pretrain' in your paper mean 'baseline,' and does 'uda' refer to 'CDTrans'?
I'm curious if 'baseline' denotes 'Deit-based' in your paper.

Compile error

I have followed the Requirements of README.md and tried to run the code for a whole day. however, there are still several issues unresolved. I searched on the Google but it did not work. I want get some advice please.

pretrain

bash scripts/pretrain/officehome/run_officehome_Ar.sh deit_small

But The Art/Pan/00002.jpg is exist in officehome dataset and Art.txt

train

bash scripts/uda/office31/run_office_amazon.sh deit_small
bash scripts/uda/visda/run_visda.sh deit_small

Question about the Loss term in this code

Hello, I think there exsists a logical error in your code.

In processor_uda.py, the loss Loss1 of the target samples is not consistant with the loss term your proposed in paper.

Here I take out the error line 339

loss1 = loss_fn(score1, feat1, t_pseudo_target, target_cam)

and the line 337

(self_score1, self_feat1, self_prob1), (score2, feat2, prob2), (score1, feat1, prob1), cross_attn = model(img, t_img, target, cam_label=target_cam, view_label=target_view ) # output: source , target , source_target_fusion

In your code, you use the cls Loss of the fusion samples instead of the target samples, then the loss is employed to optimize the network as the target cls Loss

This is not reasonable and not consistant with Fig. (2) in your paper.

Hope you can check the code. Thanks

Visualising the attention

Just wondering if you had any code to allow for visualization of the attention maps of size (B, N, num_patches+1, num_patches+1) to create something like Figure 1a in the paper.

Thanks!

for row 4 of Table 5.

您好，我有两个问题想要请教一下：

CENTER -AWARE FILTERING 过滤的是那些使用预训练模型打的伪标签与经过这个ca模块后的伪标签不一致的target domain图像吗？
有多次重复使用eq6. eq7的实验记录吗，即重复进行过滤的操作，对性能是否有进一步的提升呢？

Results are not consistent with the paper

Hi, thanks for your great work! Why are the results of DomainNet in this README not consistent with those in Table. 4 in your paper ?

CDTrans与TransReID的关系？

您好！

发现了一件古怪的事情，在本repo的代码中，似乎从来没有出现过CDTrans这一名字，反而是TransReID反复出现，请问我是否可以认为TransReID就是CDTrans的早期名字呢？

以及，请问TransReID中所需要的，并且也是经常出现的camera, view, 这些到底是什么？如何从数据集中获取它们呢？

谢谢！

find

DistributedDataParallel

Hello, can you add the operation process of multi-card training in README.md? Since I changed the DIST_TRAIN setting to True, I have encountered a series of bugs. After several changes, I am now running train.py normally, but I don't know if it is normal training.

I'm using visda + deit_base.

Problem in DomainNet training setting

Hi, why do you use combine the training set and the testing set in DomainNet for training and testing? For example, when taking "clipart" as the target domain, you use clipart.txt (combining clipart_train.txt and clipart_test.txt) for both training and testing, which means all the testing samples have been seen during training. For another word, why not use clipart_train.txt for training and use clipart_test.txt for testing?

Code problem

in processor_uda.py

line 236: train_loader, train_loader1, and train_loader2,
I don't konw train_loder1&2 means.

Single Image inference

What changes need to be made in the code as well as in the configuration file in order to carry out inference on single image.

For example in do_inference_uda

probs = model(img, img, cam_label=camids, view_label=target_view, return_logits=True)
What should the camid and target_view be set to.

Furthermore at line 530

 evaluator.update((probs[1], vid)) here vid is the class label which is being used to compute evaluate accuracy metrics.

For single image evaluation is this line required? Did you run any experiment on your end for single image classification ?

Would appreciate it if you could give some insights on this.

What different between 'pretrain' and 'UDA'?

Nice work!

bash scripts/pretrain/visda/run_visda.sh deit_base
bash scripts/uda/visda/run_visda.sh deit_base

What different between 'pretrain' and 'uda'?

用于segmentation任务？

你好，

这个模型可以被用于unsupervised domain adaptation for 3D segmentation吗？技术小白真心求教。

谢谢！

关于主流UDA数据集的训练方法

您好，感谢贵团队分享的论文及代码，受益良多。
1、首先，对于实验部分中使用的主流数据集Office31、OfficeHome以及不划分的train和test的DomainNet等，训练阶段是否都是使用source样本+target样本，在测试阶段仍然是拿训练过程中使用过的target样本？请问大致是这样的过程吗？

2、如果实验部分的确是上述第1个问题的情况，那么最终得到的模型很有可能已经过拟合target样本。同时，测试阶段用于test的样本都是网络在训练阶段见过的样本，所以我认为测试阶段得到精度无法正确评估模型在target域上的性能（或者说泛化性）。况且，CDTrans使用的是伪标签的训练方法，对带伪标签的target样本做监督学习，应该会更容易产生过拟合target样本的现象。

3、现在大部分论文的方法也都是这么使用主流数据集的吗？若上述训练和测试方式存在问题，那么那些不划分target训练和测试部分的DA方法，最终可能都是训练出来一个过拟合target样本的模型呢？即使是不使用伪标签的那些DA方法，如果将全部target样本同时用于训练和测试，是不是也是不合理的呢？

总的来讲，我认为DA最终得到的模型，应该要拿那些网络没见过的target样本去测试，才能评估这个模型是否真的适应了target的数据分布。希望作者看到的话，能帮忙看看我的问题以及观点是否有问题吗？感激不尽！😁🤞

Person Reidentification

Thank you for your great work! I notice that in your code, there are options for person re-identification training and testing. I would like to ask if you have done any work about person reid and if there are, how about the results?