christophreich1996 / cell-detr Goto Github PK

Hi Christoph,

Thank you so much for providing the code and the explanations. It is really a brilliant work.
I would like to use the Cell-DETR model for the detection and segmentation of a single object in order to study the DETR.
However, I met some problem when I prepared the train and validation data.

First step
I completed image annotations by the tool Labelme. This tool can convert to VOC-format or to COCO-format.(Labelme convert format details)

Second step
After complete annotations, I convert all the related files into a single json file with COCO json format.
(View the example json file here)

However, I'm really confused about how to prepare the data structured in bounding boxes, classes, inputs, and instances.
As for the input images, the following method may work.

import torchvision.transforms
pil_to_tensor = transforms.ToTensor()(img).unsqueeze_(0)

As for the bounding boxes, classes, and instances, should I manually edit and save as tensors in .pt files?
Could you please elaborate the above process?
Do you have any idea how to complete preparing the train and validation data in batch?
Would you please share the details how you prepare the data structured in bounding boxes, classes, inputs, and instances?
Thanks in advance.

Best regards,
Simon

Hello,

Thanks for your interesting paper and for sharing the code.

Could I ask for test method in (model_wrapper.py), there is a conversion of the bounding box coordinates format from (xc yc w h)-> (x0 y0 x1 y1) (lines 351- 354) when calculating the test metrics for the bounding boxes but why is it not the case for the validate method (lines 222-224)?

Thank you!
Charlene

Hello,I found an issue when running:
File "main.py", line 70, in
from modules.modulated_deform_conv import ModulatedDeformConvPack
ModuleNotFoundError: No module named 'modules'

Hi,
Please help me to prepare the data in .pt format.
My data is in coco format and I run maskRCNN on it.
I need to try with your model also.
Can you please support converting the data to .pt format?

Thanks
Mostafa

python setup.py build install
something error in this step

gcc -pthread -B /home/szp/anaconda3/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -DWITH_CUDA -I/home/szp/Cell-DETR/Deformable-Convolution-V2-PyTorch/src -I/home/szp/anaconda3/lib/python3.8/site-packages/torch/include -I/home/szp/anaconda3/lib/python3.8/site-packages/torch/include/torch/csrc/api/include -I/home/szp/anaconda3/lib/python3.8/site-packages/torch/include/TH -I/home/szp/anaconda3/lib/python3.8/site-packages/torch/include/THC -I/home/szp/anaconda3/include/python3.8 -c /home/szp/Cell-DETR/Deformable-Convolution-V2-PyTorch/src/cpu/deform_cpu.cpp -o build/temp.linux-x86_64-3.8/home/szp/Cell-DETR/Deformable-Convolution-V2-PyTorch/src/cpu/deform_cpu.o -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE="gcc" -DPYBIND11_STDLIB="libstdcpp" -DPYBIND11_BUILD_ABI="cxxabi1011" -DTORCH_EXTENSION_NAME=DCN -D_GLIBCXX_USE_CXX11_ABI=0 -std=c++14
cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++
gcc -pthread -B /home/szp/anaconda3/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -DWITH_CUDA -I/home/szp/Cell-DETR/Deformable-Convolution-V2-PyTorch/src -I/home/szp/anaconda3/lib/python3.8/site-packages/torch/include -I/home/szp/anaconda3/lib/python3.8/site-packages/torch/include/torch/csrc/api/include -I/home/szp/anaconda3/lib/python3.8/site-packages/torch/include/TH -I/home/szp/anaconda3/lib/python3.8/site-packages/torch/include/THC -I/home/szp/anaconda3/include/python3.8 -c /home/szp/Cell-DETR/Deformable-Convolution-V2-PyTorch/src/cpu/deform_psroi_pooling_cpu.cpp -o build/temp.linux-x86_64-3.8/home/szp/Cell-DETR/Deformable-Convolution-V2-PyTorch/src/cpu/deform_psroi_pooling_cpu.o -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE="gcc" -DPYBIND11_STDLIB="libstdcpp" -DPYBIND11_BUILD_ABI="cxxabi1011" -DTORCH_EXTENSION_NAME=DCN -D_GLIBCXX_USE_CXX11_ABI=0 -std=c++14
cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++
gcc -pthread -B /home/szp/anaconda3/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -DWITH_CUDA -I/home/szp/Cell-DETR/Deformable-Convolution-V2-PyTorch/src -I/home/szp/anaconda3/lib/python3.8/site-packages/torch/include -I/home/szp/anaconda3/lib/python3.8/site-packages/torch/include/torch/csrc/api/include -I/home/szp/anaconda3/lib/python3.8/site-packages/torch/include/TH -I/home/szp/anaconda3/lib/python3.8/site-packages/torch/include/THC -I/home/szp/anaconda3/include/python3.8 -c /home/szp/Cell-DETR/Deformable-Convolution-V2-PyTorch/src/cpu/modulated_deform_cpu.cpp -o build/temp.linux-x86_64-3.8/home/szp/Cell-DETR/Deformable-Convolution-V2-PyTorch/src/cpu/modulated_deform_cpu.o -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE="gcc" -DPYBIND11_STDLIB="libstdcpp" -DPYBIND11_BUILD_ABI="cxxabi1011" -DTORCH_EXTENSION_NAME=DCN -D_GLIBCXX_USE_CXX11_ABI=0 -std=c++14
cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++
/usr/bin/nvcc -DWITH_CUDA -I/home/szp/Cell-DETR/Deformable-Convolution-V2-PyTorch/src -I/home/szp/anaconda3/lib/python3.8/site-packages/torch/include -I/home/szp/anaconda3/lib/python3.8/site-packages/torch/include/torch/csrc/api/include -I/home/szp/anaconda3/lib/python3.8/site-packages/torch/include/TH -I/home/szp/anaconda3/lib/python3.8/site-packages/torch/include/THC -I/home/szp/anaconda3/include/python3.8 -c /home/szp/Cell-DETR/Deformable-Convolution-V2-PyTorch/src/cuda/deform_psroi_pooling_cuda.cu -o build/temp.linux-x86_64-3.8/home/szp/Cell-DETR/Deformable-Convolution-V2-PyTorch/src/cuda/deform_psroi_pooling_cuda.o -D__CUDA_NO_HALF_OPERATORS -D__CUDA_NO_HALF_CONVERSIONS -D__CUDA_NO_HALF2_OPERATORS --expt-relaxed-constexpr --compiler-options '-fPIC' -DCUDA_HAS_FP16=1 -D__CUDA_NO_HALF_OPERATORS -D__CUDA_NO_HALF_CONVERSIONS_ -D__CUDA_NO_HALF2_OPERATORS__ -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE="_gcc" -DPYBIND11_STDLIB="_libstdcpp" -DPYBIND11_BUILD_ABI="_cxxabi1011" -DTORCH_EXTENSION_NAME=DCN -D_GLIBCXX_USE_CXX11_ABI=0 -gencode=arch=compute_75,code=sm_75 -std=c++14
nvcc fatal : Unsupported gpu architecture 'compute_75'
error: command '/usr/bin/nvcc' failed with exit status 1

Hello,
I don't understand if not ohem, why do you index with [:prediction.shape[-1]]? Shouldn't it be the reverse where if ohem is True, then the batch size is reduced?

Hi, I want to ask how many GPUs are used in this Cell-DETR?

wanna get the dataset, but the link in readme to ask for the dataset is invalid.
Hope to access the dataset thanks

Hello, when I was following your guidance to do tests on the functions of the network, I met such a problem:


It seems that we still need the data for training when applying the trained models. So are there any problems with our test, or we should download the whole dataset before conducting this test?
Meanwhile, since all of the experiments in your essay were conducted on images of 128128 pixels, however, the images we achieved in our experiments had a size of 20482048 pixels, which were much larger than those of the dataset. When adopting Cell-DETR to our further research, are the images too large for the network to deal with, as the complexity of a Transformer corresponds to O(n^2d)? Were there any tests or experiments on the images with larger sizes?
Thank you!

Coutld you share the download link of the data set?

from modules.modulated_deform_conv import ModulatedDeformConvPack

I wasn't able to import this package. I tried build from [https://github.com/chengdazhi/Deformable-Convolution-V2-PyTorch]
and the modules used to build DCN is deprecated.
Please help with this issue and it would be nice if you upload that modules directory along with latest stable build script for DCN.

Epoch 5/10 Best val. mIoU=0.0000 Loss C.=0.2363 Loss BB.=0.4493 Loss Seg.=1.7465: 50% 1480/2960 [01:15<01:05, 22.48it/s]/usr/local/lib/python3.8/site-packages/numpy/core/fromnumeric.py:3432: RuntimeWarning: Mean of empty slice.
return _methods._mean(a, axis=axis, dtype=dtype,
/usr/local/lib/python3.8/site-packages/numpy/core/_methods.py:190: RuntimeWarning: invalid value encountered in double_scalars
ret = ret.dtype.type(ret / rcount)
Epoch 10/10 Best val. mIoU=0.0000 Loss C.=0.2336 Loss BB.=0.4569 Loss Seg.=1.2564: 100% 2960/2960 [02:31<00:00, 23.09it/s]/usr/local/lib/python3.8/site-packages/numpy/core/fromnumeric.py:3432: RuntimeWarning: Mean of empty slice.
return _methods._mean(a, axis=axis, dtype=dtype,
/usr/local/lib/python3.8/site-packages/numpy/core/_methods.py:190: RuntimeWarning: invalid value encountered in double_scalars
ret = ret.dtype.type(ret / rcount)
Epoch 10/10 Best val. mIoU=0.0000 Loss C.=0.2336 Loss BB.=0.4569 Loss Seg.=1.2564: 100% 2960/2960 [02:40<00:00, 23.09it/s]/usr/local/lib/python3.8/site-packages/numpy/core/fromnumeric.py:3432: RuntimeWarning: Mean of empty slice.
return _methods._mean(a, axis=axis, dtype=dtype,
/usr/local/lib/python3.8/site-packages/numpy/core/_methods.py:190: RuntimeWarning: invalid value encountered in double_scalars
ret = ret.dtype.type(ret / rcount)
Epoch 10/10 Best val. mIoU=0.0000 Loss C.=0.2336 Loss BB.=0.4569 Loss Seg.=1.2564: 100% 2960/2960 [02:41<00:00, 18.29it/s]
/usr/local/lib/python3.8/site-packages/numpy/core/fromnumeric.py:3432: RuntimeWarning: Mean of empty slice.
return _methods._mean(a, axis=axis, dtype=dtype,

On the last line of the installation code
(venv) maxs@doraemon1:~/Cell-DETR/pade_activation_unit/cuda$ python setup.py build install

I get:
Traceback (most recent call last): File "setup.py", line 301, in <module> generate_cpp_module(fname='pau_cuda.cpp') File "setup.py", line 51, in generate_cpp_module content = file_content.merge(locals()) File "/home/maxs/.virtualenvs/venv/lib/python3.6/site-packages/airspeed/__init__.py", line 95, in merge self.merge_to(namespace, output, loader) File "/home/maxs/.virtualenvs/venv/lib/python3.6/site-packages/airspeed/__init__.py", line 105, in merge_to self.ensure_compiled() File "/home/maxs/.virtualenvs/venv/lib/python3.6/site-packages/airspeed/__init__.py", line 100, in ensure_compiled self.root_element = TemplateBody(self.filename, self.content) File "/home/maxs/.virtualenvs/venv/lib/python3.6/site-packages/airspeed/__init__.py", line 270, in __init__ self.parse() File "/home/maxs/.virtualenvs/venv/lib/python3.6/site-packages/airspeed/__init__.py", line 1245, in parse self.block = self.next_element(Block) File "/home/maxs/.virtualenvs/venv/lib/python3.6/site-packages/airspeed/__init__.py", line 314, in next_element element = element_spec(self.filename, self._full_text, self.end) File "/home/maxs/.virtualenvs/venv/lib/python3.6/site-packages/airspeed/__init__.py", line 270, in __init__ self.parse() File "/home/maxs/.virtualenvs/venv/lib/python3.6/site-packages/airspeed/__init__.py", line 1277, in parse FallthroughHashText))) File "/home/maxs/.virtualenvs/venv/lib/python3.6/site-packages/airspeed/__init__.py", line 321, in next_element self.end) File "/home/maxs/.virtualenvs/venv/lib/python3.6/site-packages/airspeed/__init__.py", line 270, in __init__ self.parse() File "/home/maxs/.virtualenvs/venv/lib/python3.6/site-packages/airspeed/__init__.py", line 1033, in parse self.require_match(self.OPEN_PAREN, '(') File "/home/maxs/.virtualenvs/venv/lib/python3.6/site-packages/airspeed/__init__.py", line 308, in require_match raise self.syntax_error(expected) airspeed.TemplateSyntaxError: line 6, column 8: expected ( in define definition, got: CHECK_CUDA(x) AT_ASSERTM(x.type().i ... ...

I thought the issue was with the Airspeed library / cuda combination, but I've updated both. The setup.py file has an airspeed template that doesn't match the expectations of airspeed itself. Have you any idea where I should look for the problem? I can't find a single thing about this problem anywhere. Do I need a specific older version of Airspeed?

Thanks in advance for reading this message,
Max

Hi, I would like to ask if I can set the weight of segmentation loss and classification loss to 0?

My own dataset don't have segmentation label, and only have 1 class

does it work on your model?

problem solved

Traceback (most recent call last):
File "main.py", line 150, in
model_wrapper.test()
File "/usr/local/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 49, in decorate_no_grad
return func(*args, **kwargs)
File "/content/drive/MyDrive/Cell-DETR/model_wrapper.py", line 421, in test
misc.plot_instance_segmentation_labels(
File "/content/drive/MyDrive/Cell-DETR/misc.py", line 435, in plot_instance_segmentation_labels
instances_map += np.array(colors_traps[min(counter_trap_instance, len(colors_cells) - 1)]).reshape(1, 1, 3)
IndexError: tuple index out of range

I failed at setup.py in cuda/

error message:

gcc-6.3.0: No such file or directory
nvcc fatal : Failed to preprocess host compiler properties.
error: command '/usr/local/cuda/bin/nvcc' failed with exit status 1

Hi there,
first of all thank you so much for providing the code to your paper and for the clear explanations.

I want to use the Cell-DETR model for a use-case with more than 20 labels per image.
I already got everything running, but it seems like that the model only predicts 12 bounding boxes and masks and the predicted boxes are also all located on one side of the image.

Do you have any idea where that might come from? Is there an additional hardcoded parameter in the architecture that I need to change?

Best,
Lisa