gau-nernst / vision-toolbox Goto Github PK

Hey,
I've been trying to load the pre-trained weights from this repo into the original YOLOv5 (changing state dict key names, etc.).
However, something does not align in the weights shapes, for example on YOLOv5s:
(left - your implementation with names and sizes, right - original YOLOv5)

The first problem occurs In block 0 - a mismatch in the weights shapes as shown in the image above.
I've checked it on the s,m,l , all of them having the same issue.

Could you please check if there's a bug in your implementation?
Thanks!

For learning purpose

• ViT: https://arxiv.org/abs/2010.11929
• MLP-Mixer: https://arxiv.org/abs/2105.01601
• DeiT: https://arxiv.org/abs/2012.12877
• DeiT-III: https://arxiv.org/abs/2204.07118
• CaiT: https://arxiv.org/abs/2103.17239
• Swin: https://arxiv.org/abs/2103.14030
• Swin V2: https://arxiv.org/abs/2111.09883
• MobileViT: https://arxiv.org/abs/2110.02178
• MobileViTv2: https://arxiv.org/abs/2206.02680
• MobileViTv3: https://arxiv.org/abs/2209.15159
• ConvNeXt: https://arxiv.org/abs/2201.03545
• ConvNeXt-V2: https://arxiv.org/abs/2301.00808
• TinyViT: https://arxiv.org/abs/2207.10666
• MobileOne: https://arxiv.org/abs/2206.04040
• EfficientViT: https://arxiv.org/abs/2305.07027
• GC ViT: https://arxiv.org/abs/2206.09959
• Fast ViT: https://arxiv.org/abs/2303.14189
• Faster ViT: https://arxiv.org/abs/2306.06189
• MaxViT: https://arxiv.org/abs/2204.01697

Probably will port weights.

🚀 Features and Motivation

Hi @gau-nernst , This toolbox looks very impressive!

There are several derivative versions of darknet in the practical application of object detection, such as the popular yolov5 and yolox, which modify darknet a bit. Do you have any plans to support these versions?

Additional context

• https://github.com/ultralytics/yolov5/blob/436ffc4/models/yolov5l.yaml#L13-L25
• https://github.com/Megvii-BaseDetection/YOLOX/blob/main/yolox/models/darknet.py
• https://github.com/zhiqwang/yolov5-rt-stack/blob/main/yolort/models/darknetv6.py (This is a refactored version of YOLOv5 Darknet to make it readable in the torchvision layout)

I've seen your instructions to train on Imagenet Object Localization.

How long does it take to train on such a big dataset?

Could you please share a trained checkpoint for Yolov5 and any other architecture that you've tested?

It would be great for people with less resources and to save a little bit of energy ;-)

If it wouldn't be possible for any reason, it's perfectly understandable. Thanks for sharing this project

hey,dude.I find you are the only one to train imagenet in yolov5.
I wanna to use your pretrained model in yolov5 for object detecting.
But it cann't take it in raw yolov5?

python train.py --data tinyObject.yaml --cfg ./models/yolov5m.yaml  --weights darknet_yolov5m-a1eb31bd.pt

error:  File "train.py", line 123, in train
    csd = ckpt['model'].float().state_dict()  # checkpoint state_dict as FP32
KeyError: 'model'

So what can I do to edit backbone net?

What happened to Timm backbones?
Before the refactor to vision toolbox, I was using timm_dla34 for example.

As per title.

TODO

Backbones

• HRNet [Paper v1] [Paper v2] [code] [mmdet]
• VOLO? [paper]
• PatchConvNet (training results are not good. port weights over to check correctness?)
• VoVNetV1, since ESE is not supported on many edge platforms
• torchvision's RegNet [paper] [code]
• Darknet-YOLOv5 [code]

Training

• (Low priority) JPEG GPU decoding to alleviate CPU bottleneck in data loading for small models link
  • May break torchvision.transforms and Lightning
• (Low priority) Model EMA
  • Lightning-AI/pytorch-lightning/issues/10914 -> not a clean solution
  • torchvision
• Use DataModule to separate data from LightningModule for better readability
• Channels-last memory layout link
  • Good performance gain (RTX3090, with fp16 training). For Darknet-53, 1.4 it/s (w/ JIT) -> 2 it/s (w/o JIT, channels-last)
  • JIT + channels-last is not good: 1.3it/s
• WebDataset
• (Currently not possible) DALI pipeline
  • NVIDIA/DALI/issues/2978
  • DALI does not support conditional operations. Implementing Trivial Augment will be challenging