Code Monkey home page Code Monkey logo

deformable-detr-'s Introduction

Deformable DETR for custom dataset

By Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, Jifeng Dai.

This repository is a fork of official implementation of the paper Deformable DETR: Deformable Transformers for End-to-End Object Detection. Some small adjusts were made in the files to be able to train any custom dataset, according to what is made on the original DETR and on Deformable-DETR.

Introduction

TL; DR. Deformable DETR is an efficient and fast-converging end-to-end object detector. It mitigates the high complexity and slow convergence issues of DETR via a novel sampling-based efficient attention mechanism.

deformable_detr

deformable_detr

Abstract. DETR has been recently proposed to eliminate the need for many hand-designed components in object detection while demonstrating good performance. However, it suffers from slow convergence and limited feature spatial resolution, due to the limitation of Transformer attention modules in processing image feature maps. To mitigate these issues, we proposed Deformable DETR, whose attention modules only attend to a small set of key sampling points around a reference. Deformable DETR can achieve better performance than DETR (especially on small objects) with 10× less training epochs. Extensive experiments on the COCO benchmark demonstrate the effectiveness of our approach.

License

This project is released under the Apache 2.0 license.

Changelog

See changelog.md for detailed logs of major changes.

Citing Deformable DETR

If you find Deformable DETR useful in your research, please consider citing:

@article{zhu2020deformable,
  title={Deformable DETR: Deformable Transformers for End-to-End Object Detection},
  author={Zhu, Xizhou and Su, Weijie and Lu, Lewei and Li, Bin and Wang, Xiaogang and Dai, Jifeng},
  journal={arXiv preprint arXiv:2010.04159},
  year={2020}
}

Main Results

Method Epochs AP APS APM APL params
(M)
FLOPs
(G)
Total
Train
Time
(GPU
hours)
Train
Speed
(GPU
hours
/epoch)
Infer
Speed
(FPS)
Batch
Infer
Speed
(FPS)
URL
Faster R-CNN + FPN 109 42.0 26.6 45.4 53.4 42 180 380 3.5 25.6 28.0 -
DETR 500 42.0 20.5 45.8 61.1 41 86 2000 4.0 27.0 38.3 -
DETR-DC5 500 43.3 22.5 47.3 61.1 41 187 7000 14.0 11.4 12.4 -
DETR-DC5 50 35.3 15.2 37.5 53.6 41 187 700 14.0 11.4 12.4 -
DETR-DC5+ 50 36.2 16.3 39.2 53.9 41 187 700 14.0 11.4 12.4 -
Deformable DETR
(single scale)
50 39.4 20.6 43.0 55.5 34 78 160 3.2 27.0 42.4 config
log
model
Deformable DETR
(single scale, DC5)
50 41.5 24.1 45.3 56.0 34 128 215 4.3 22.1 29.4 config
log
model
Deformable DETR 50 44.5 27.1 47.6 59.6 40 173 325 6.5 15.0 19.4 config
log
model
+ iterative bounding box refinement 50 46.2 28.3 49.2 61.5 41 173 325 6.5 15.0 19.4 config
log
model
++ two-stage Deformable DETR 50 46.9 29.6 50.1 61.6 41 173 340 6.8 14.5 18.8 config
log
model

Note:

  1. All models of Deformable DETR are trained with total batch size of 32.
  2. Training and inference speed are measured on NVIDIA Tesla V100 GPU.
  3. "Deformable DETR (single scale)" means only using res5 feature map (of stride 32) as input feature maps for Deformable Transformer Encoder.
  4. "DC5" means removing the stride in C5 stage of ResNet and add a dilation of 2 instead.
  5. "DETR-DC5+" indicates DETR-DC5 with some modifications, including using Focal Loss for bounding box classification and increasing number of object queries to 300.
  6. "Batch Infer Speed" refer to inference with batch size = 4 to maximize GPU utilization.
  7. The original implementation is based on our internal codebase. There are slight differences in the final accuracy and running time due to the plenty details in platform switch.

Installation

Requirements

  • Linux, CUDA>=9.2, GCC>=5.4

  • Python>=3.7

    We recommend you to use Anaconda to create a conda environment:

    conda create -n deformable_detr python=3.7 pip

    Then, activate the environment:

    conda activate deformable_detr
  • PyTorch>=1.5.1, torchvision>=0.6.1 (following instructions here)

    For example, if your CUDA version is 9.2, you could install pytorch and torchvision as following:

    conda install pytorch=1.5.1 torchvision=0.6.1 cudatoolkit=9.2 -c pytorch
  • Other requirements

    pip install -r requirements.txt
  • To run without issues you must run (with exclamation on Colab)

    !pip install torch==1.8.0 torchvision==0.9.0 -qq
    

Edit 11/12/2021

  • If you are running on a GPU A100-SXM4-40GB that would be better to use the 1.8 version of torch
!pip install torch==1.8.0+cu111 torchvision==0.9.0+cu111 torchaudio==0.8.0 -f https://download.pytorch.org/whl/torch_stable.html

Compiling CUDA operators

cd ./models/ops
sh ./make.sh
# unit test (should see all checking is True)
python test.py
  • After running the previous commands you must restart your colab

Usage

Dataset preparation

Please download COCO 2017 dataset and organize them as following:

code_root/
└── data/
    └── coco/
        ├── train2017/
        ├── val2017/
        └── annotations/
        	├── instances_train2017.json
        	└── instances_val2017.json
  • Before start to train adjust the number of classes on deformable_detr.py to your number of classes:
  445  #num_classes = 20 if args.dataset_file != 'coco' else 91
  446  #if args.dataset_file == "coco_panoptic":
  447  #    num_classes = 250
    
       num_classes = your_number_of_classes + 1

Training

Training on single node

For example, the command for training Deformable DETR on 8 GPUs is as following:

GPUS_PER_NODE=8 ./tools/run_dist_launch.sh 8 ./configs/r50_deformable_detr.sh

Training on multiple nodes

For example, the command for training Deformable DETR on 2 nodes of each with 8 GPUs is as following:

On node 1:

MASTER_ADDR=<IP address of node 1> NODE_RANK=0 GPUS_PER_NODE=8 ./tools/run_dist_launch.sh 16 ./configs/r50_deformable_detr.sh

On node 2:

MASTER_ADDR=<IP address of node 1> NODE_RANK=1 GPUS_PER_NODE=8 ./tools/run_dist_launch.sh 16 ./configs/r50_deformable_detr.sh

Training on single-GPU

In the situation you are using just one GPU you must run the following line, for example (On colab use the exclamation):

!python main.py --dataset_file custom --data_path /content/coco --output_dir /content/Output --batch_size 3 --epochs 10

Training on slurm cluster

If you are using slurm cluster, you can simply run the following command to train on 1 node with 8 GPUs:

GPUS_PER_NODE=8 ./tools/run_dist_slurm.sh <partition> deformable_detr 8 configs/r50_deformable_detr.sh

Or 2 nodes of each with 8 GPUs:

GPUS_PER_NODE=8 ./tools/run_dist_slurm.sh <partition> deformable_detr 16 configs/r50_deformable_detr.sh

Some tips to speed-up training

  • If your file system is slow to read images, you may consider enabling '--cache_mode' option to load whole dataset into memory at the beginning of training.
  • You may increase the batch size to maximize the GPU utilization, according to GPU memory of yours, e.g., set '--batch_size 3' or '--batch_size 4'.

Evaluation

You can get the config file and pretrained model of Deformable DETR (the link is in "Main Results" session), then run following command to evaluate it on COCO 2017 validation set:

<path to config file> --resume <path to pre-trained model> --eval

You can also run distributed evaluation by using ./tools/run_dist_launch.sh or ./tools/run_dist_slurm.sh.

Inference

For getting inference you may run the inference.py for getting the bounding boxes for a given image, different of DETR that apply softmax() here is applied sigmoid() and some scores might be out of the scale from 0 to 1. You could customize the file to find your specific image path.

python inference.py --resume /path/to/checkpoint.pth 

deformable-detr-'s People

Contributors

ver0z avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar

deformable-detr-'s Issues

RuntimeError: The size of tensor a (91) must match the size of tensor b (7) at non-singleton dimension 0

When I try to train custom dataset, the following error occurs. After start training it will occurs. Could you please tell me where was wrong?

Start training
Traceback (most recent call last):
File "main.py", line 327, in
main(args)
File "main.py", line 277, in main
model, criterion, data_loader_train, optimizer, device, epoch, args.clip_max_norm)
File "/mnt/e/opencv/Deformable-DETR--main1/Deformable-DETR--main/engine.py", line 56, in train_one_epoch
optimizer.step()
File "/home/bugs/anaconda3/envs/deformable_detr/lib/python3.7/site-packages/torch/optim/lr_scheduler.py", line 67, in wrapper
return wrapped(*args, **kwargs)
File "/home/bugs/anaconda3/envs/deformable_detr/lib/python3.7/site-packages/torch/autograd/grad_mode.py", line 15, in decorate_context
return func(*args, **kwargs)
File "/home/bugs/anaconda3/envs/deformable_detr/lib/python3.7/site-packages/torch/optim/adamw.py", line 104, in step
exp_avg.mul_(beta1).add_(grad, alpha=1 - beta1)
RuntimeError: The size of tensor a (91) must match the size of tensor b (7) at non-singleton dimension 0

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.