suyz526 / zebrapose Goto Github PK
View Code? Open in Web Editor NEWThe implementation of the paper 'ZebraPose: Coarse to Fine Surface Encoding for 6DoF Object Pose Estimation' (CVPR2022)
License: MIT License
The implementation of the paper 'ZebraPose: Coarse to Fine Surface Encoding for 6DoF Object Pose Estimation' (CVPR2022)
License: MIT License
Hi,
Thanks for your great job and your kindness in sharing your pretrained models!
I'm wondering if you could share the training details of other datasets in BOP besides YCBV and LM/LMO. Such as the choice of training data, the augmentation skills, etc.
It would be really helpful for reproducing your results and develop your method.
Hi,
I have read your paper and the idea is really interesting! But I am not sure whether you train separate networks for each object (i.e. P.E.=N
defined in GDR-Net) or train one unified network for all objects (i.e. P.E.=1
). Could you help to clarify this? Thanks!
Also, in my understanding, the results of GDR-Net are P.E.=N
while those of SO-Pose are P.E.=1
.Since P.E.=N
tends to outperform P.E.=1
for the same method, putting these results together without additional remarks may look a bit weird.
Hi,
Thank you for your work. I would like to try to train with my own dataset but I can't seem to understand what are the "Class_CorresPoint0000XX.txt". How do you obtain them and what do they represent?
Thank you for your response.
Dear authors,
Thank you for your contribution and work.
I am trying to run your code for my own research. However, CUDA keeps running out of memory.
I have tried to make the following modifications, some have some effect of the memory running out a bit slower:
I am trying to train on a small subset of TUD-L dataset, using the GTs and pre-trained backbones provided through the links on this repo.
Here is a screenshot from the Colab environment:
Suddenly, there is a huge spike in GPU RAM usage, seen on the right in the dashboard.
Is this an issue or is this expected behaviour? Your help would be much appreciated.
Best regards,
Thijs
Hello, I would like to ask where the implementation for loss curves and visualization of pose estimation results is located?
when i try to run python scripts/vis_est_poses.py,it encountered this error.
I looked at the code, and there are options to choose between 'flat' and 'phong'. It seems that 'flat' may not require normals, but 'phong' appears to provide better visual effects.
How would you like to address this issue?
Do you prepare to release your results (eg. add-s) on other datasets like tless?
Hi all, I was trying to re-run your baseline results using the provided checkpoints on the LMO dataset. What I found is that the results were closer to V1 of your paper. I even tried to change the resize methods in inference but it's not giving me the same results. Can someone help me on this?
Hi,
If I want to compare the entire mask to obtain a loss function, what part of the code do I need to modify? Or do I need to retrain the ResNet34 model?
if predict_entire_mask:
loss_m = maskLoss(pred_mask_prob, entire_masks)
else:
loss_m = maskLoss(pred_mask_prob, masks)
Best,
Joa
Hi,
I find out that you have uploaded a new version of your paper in arXiv, and the performances are greatly improved compared with v1. However, I could not figure out what contributed to such significant improvement. Could you help clarify this? Thanks very much!
Hi,
I want to ask, after the Fcos detector obtains the results of Bboxes, how does the ResNet-34 network extract features from the ROI region of interest(e.g. ape)? Is this implemented by the pre-trained model in the pretrained_backbone
folder?
Best,
Joa
Hi, thanks for your great work! I wonder how can I download files from DFKI cloud directly via commands like wget? Thanks
Hi Yongzhi,
I couldnt find a score for prediction and refinement in test_vivo.py . I think I am again missing something there. How do we calculate the score for both predictions and icp refinements?
Thanks in advance
Hi,
Thank you for your great work. I'm very interested in your proposed method and wanna reproduce the results on LM-O, but it seems that you didn't provide pretrained weight of each object. It would be very kind if you make the pretrained weights publicly available.
Best regards
Leroy
Hi suyz526,
Thanks, thanks a lot work your great work. I have big hopes with our dataset for your project if I can make it work of course. I have been working for 3 days to make it work. Especially render module became a torture for me : )
Anyway, I am very close to see the refined positions on renderer but I am getting error like below
AttributeError: module 'cv2' has no attribute 'ppf_match_3d_ICP'
on code line 129 on icp_refiner_modified.py.
icp_fnc = cv.ppf_match_3d_ICP(n_iters, tolerence=tolerence, numLevels=4)
I am misssing something but I couldnt figure out.
I have installed the below modules:
pip install opencv-python==4.5.4.58
pip install opencv-contrib-python==4.5.4.60
but I am still having the same problem. cv2 cannot find surface matching module and ppf.
Do you have any suggestion for this problem?
thanks thanks in advance
Hi all,
I'm currently trying to apply Zebrapose to my own dataset. I've sucessfully generated the c++ executable and the python wrapper, and I can run the generator to render the zebrapose annotated images
python3 generate_training_labels_for_BOP.py --bop_path /home/user/datasets/bop/ --dataset_name myDataset --force_rewrite True --is_training_data False --data_folder test --start_obj_id 0 --end_obj_id 1
The script runs fine and I can see that correct images are being generated, see the following screenshot:
However, when I open the saved images, all of them have some artifacts on the top border, and the rendered image content is shifted by the same height along the y axis. See this example:
My specs:
Has anyone experienced a similar issue or a good guess what could be the issue? Thanks in advance!
Dear Authors,
I hope this message finds you well. I am writing to express my appreciation for your remarkable work. I have been utilizing your pretrained weights of LM-O in an attempt to reproduce the results presented in your paper. My results closely align with yours, with a minor difference (75.4 vs. 76.9).
While examining your scripts, I noticed the argument “--use_icp True”. I was intrigued by the potential impact of using ICP to refine the pose. However, when I incorporated “--use_icp True” during evaluation, I observed a significant drop in the results ( can
, driller
, glue
). I am curious to know if you have previously encountered this issue and if you could possibly shed some light on why this might be happening.
Here are the results for your reference:
Method | Ape | Can | Cat | Driller | Duck | Eggbox | Glue | HolePuncher | Mean |
---|---|---|---|---|---|---|---|---|---|
ZebraPose(in your paper) | 57.9 | 95.0 | 60.6 | 94.8 | 64.5 | 70.9 | 88.7 | 83.0 | 76.9 |
ZebraPose(reproduced) | 55.0 | 94.5 | 57.9 | 93.2 | 61.7 | 70.6 | 89.0 | 81.2 | 75.4 |
ZebraPose+ICP(reproduced) | 60.4 | 57.4 | 62.1 | 62.2 | 63.8 | 73.8 | 69.9 | 88.4 | 68.2 |
Yours,
Leroy Chou
I want to know how long to train one object, and do you train all object in total iteration 380000? And do you train on one GPU.
Hello, I want to ask Where can I obtain the JSON file of the 2D object detection results for GDRNPPDet?
Hi thanks for sharing the code. In terms of the reported results of Ycbv dataset in the paper, I think the reported AUC_ADD(-S)/AUC_ADD-S corresponds to the AUC_PoseCNN_ADD/ADI in the code. I want to confirm whether my understanding is correct or not. Thanks!
When building "Binary_Code_GT_Generator/Generate_Mesh_with_GT_Color" i received a lot of errors.
I fixed the issue by changing the required c++ version from "-std=c++11" to "-std=c++14" In this cmake file https://github.com/suyz526/ZebraPose/blob/main/Binary_Code_GT_Generator/Generate_Mesh_with_GT_Color/CMakeLists.txt
Is this correct?
Hi, I try to run python scripts/vis_est_poses.py
for the visualization of pose estimation results. It ran successfully, but the outputs images don't show the pose estimation:
(zebrapose) simoc@IITICB002LW012:~/ZebraPose/bop_toolkit$ python scripts/vis_est_poses.py
3/1|13:06:11: Processing: zebrapose_lmo-test.csv
vispy uses app: egl, gl: gl2
3/1|13:06:12: Loading 3D model of object 1...
3/1|13:06:12: Loading 3D model of object 5...
3/1|13:06:12: Loading 3D model of object 6...
3/1|13:06:12: Loading 3D model of object 8...
3/1|13:06:12: Loading 3D model of object 9...
3/1|13:06:12: Loading 3D model of object 10...
3/1|13:06:12: Loading 3D model of object 11...
3/1|13:06:12: Loading 3D model of object 12...
3/1|13:06:12: Loading pose estimates...
3/1|13:06:12: Organizing pose estimates...
3/1|13:06:13: Visualizing pose estimates - method: zebrapose, dataset: lmo, scene: 2, im: 3
WARNING: Value provided for 'a_normal', but this variable was not found in the shader program.
WARNING: Value provided for 'a_texcoord', but this variable was not found in the shader program.
WARNING: Value provided for 'a_normal', but this variable was not found in the shader program.
WARNING: Value provided for 'a_texcoord', but this variable was not found in the shader program.
WARNING: Value provided for 'a_normal', but this variable was not found in the shader program.
WARNING: Value provided for 'a_texcoord', but this variable was not found in the shader program.
WARNING: Value provided for 'a_normal', but this variable was not found in the shader program.
WARNING: Value provided for 'a_texcoord', but this variable was not found in the shader program.
WARNING: Value provided for 'a_normal', but this variable was not found in the shader program.
WARNING: Value provided for 'a_texcoord', but this variable was not found in the shader program.
WARNING: Value provided for 'a_normal', but this variable was not found in the shader program.
WARNING: Value provided for 'a_texcoord', but this variable was not found in the shader program.
WARNING: Value provided for 'a_normal', but this variable was not found in the shader program.
WARNING: Value provided for 'a_texcoord', but this variable was not found in the shader program.
WARNING: Value provided for 'a_normal', but this variable was not found in the shader program.
WARNING: Value provided for 'a_texcoord', but this variable was not found in the shader program.
3/1|13:06:16: Visualizing pose estimates - method: zebrapose, dataset: lmo, scene: 2, im: 62
3/1|13:06:19: Visualizing pose estimates - method: zebrapose, dataset: lmo, scene: 2, im: 107
3/1|13:06:22: Visualizing pose estimates - method: zebrapose, dataset: lmo, scene: 2, im: 166
3/1|13:06:25: Visualizing pose estimates - method: zebrapose, dataset: lmo, scene: 2, im: 254
3/1|13:06:28: Visualizing pose estimates - method: zebrapose, dataset: lmo, scene: 2, im: 342
3/1|13:06:31: Visualizing pose estimates - method: zebrapose, dataset: lmo, scene: 2, im: 434
3/1|13:06:34: Visualizing pose estimates - method: zebrapose, dataset: lmo, scene: 2, im: 480
3/1|13:06:37: Visualizing pose estimates - method: zebrapose, dataset: lmo, scene: 2, im: 549
3/1|13:06:40: Visualizing pose estimates - method: zebrapose, dataset: lmo, scene: 2, im: 649
3/1|13:06:43: Visualizing pose estimates - method: zebrapose, dataset: lmo, scene: 2, im: 737
3/1|13:06:47: Visualizing pose estimates - method: zebrapose, dataset: lmo, scene: 2, im: 770
3/1|13:06:50: Visualizing pose estimates - method: zebrapose, dataset: lmo, scene: 2, im: 809
3/1|13:06:53: Visualizing pose estimates - method: zebrapose, dataset: lmo, scene: 2, im: 883
3/1|13:06:56: Visualizing pose estimates - method: zebrapose, dataset: lmo, scene: 2, im: 942
3/1|13:06:58: Visualizing pose estimates - method: zebrapose, dataset: lmo, scene: 2, im: 1038
3/1|13:07:02: Visualizing pose estimates - method: zebrapose, dataset: lmo, scene: 2, im: 1107
3/1|13:07:05: Visualizing pose estimates - method: zebrapose, dataset: lmo, scene: 2, im: 178
3/1|13:07:07: Visualizing pose estimates - method: zebrapose, dataset: lmo, scene: 2, im: 611
3/1|13:07:10: Visualizing pose estimates - method: zebrapose, dataset: lmo, scene: 2, im: 840
3/1|13:07:12: Done.
In this issue, I have seen that @hrkita produced some results, so, am I missing something?
Thanks,
Simone
Dear Yongzhi,
I have compiled and installed the GT generation part of the code and dependencies.
The GT object mesh is successfully generated, but I am running into an error when trying to render the GT training data.
I changed the line 39 in the opengl_render.cpp to my local path to the render_related_source folder.
It happens on both my own dataset and on LM-O.
I also noticed that this portion of the code received major updates 15 days ago. Is this a working version on your system?
Below is the resulting output.
$ python generate_training_labels_for_BOP.py --bop_path $BOP_PATH --dataset_name lmo --force_rewrite True --is_training_data True --data_folder train --start_obj_id 1 --end_obj_id 1
/media/pmvanderburg/T71/bop_datasets/lmo/models/obj_000001.ply
/media/pmvanderburg/T71/bop_datasets/lmo/models/obj_000005.ply
/media/pmvanderburg/T71/bop_datasets/lmo/models/obj_000006.ply
/media/pmvanderburg/T71/bop_datasets/lmo/models/obj_000008.ply
/media/pmvanderburg/T71/bop_datasets/lmo/models/obj_000009.ply
/media/pmvanderburg/T71/bop_datasets/lmo/models/obj_000010.ply
/media/pmvanderburg/T71/bop_datasets/lmo/models/obj_000011.ply
/media/pmvanderburg/T71/bop_datasets/lmo/models/obj_000012.ply
if models are not fully listed above, please make sure there are ply files available
[640. 480. 572.4114 573.57043 325.2611 242.04899]
ERROR::SHADER::FILE_NOT_SUCCESFULLY_READ
ERROR::SHADER_COMPILATION_ERROR of type: VERTEX
0:1(1): error: syntax error, unexpected $end
-- --------------------------------------------------- --
ERROR::SHADER_COMPILATION_ERROR of type: FRAGMENT
0:1(1): error: syntax error, unexpected $end
-- --------------------------------------------------- --
ERROR::PROGRAM_LINKING_ERROR of type: PROGRAM
error: linking with uncompiled/unspecialized shadererror: linking with uncompiled/unspecialized shader
-- --------------------------------------------------- --
Can you maybe point me in the right direction?
Best,
Thijs
First, thanks for the wonderful paper. It was a pleasure to read and I look forward to working with the code to understand the approach better.
Can you kindly explain again how to download the pre-trained models? The README links to https://cloud.dfki.de/owncloud/index.php/s/EmQDWgd5ipbdw3E. For example the ape of lmo folder contains a version file, a data.pkl file, and a subfolder data
which contains several files but it is not clear to me that the checkpoints are among them.
Hi,
Thanks again for sharing your great work! I am a bit confused about the bop version and paper version of YCB-V dataset. The configs looks the same, except the option bop_challange
. Does bop_challange
only affect the test data? Thanks!
Hi!
Thank you for your amazing work!
I have some questions about the parameter "efficientnet_key", what is the purpose of this parameter? Is it to improve the model's inference time and reduce GPU memory usage? In your code, you have implemented two options, 'b5' and 'b4'. What do they represent respectively? Does using this parameter lead to a decrease in the accuracy of model inference?
how many pics(come from PBR) do you use when training LMO dataset? all of them or the part that contain the object
As the title says i am having some performance issues when training.
training 1000 iterations takes on average 60 minutes. Of these 60min around 50min are executing test_network_with_single_obj
. The weird thing is that sometimes test_network_with_single_obj
will finish within 20 seconds
I have tried training on my own dataset and lmo the problem persists with both.
The computer specs are 2080 ti and a 9900k, 32GB of ram
I have tried different combinations of batch_size
and num_workers
.
progressive X has also been disabled.
Any other suggestions as to what might cause this issue?
Hello. I am trying out zebrapose and I'm having some issues when training on the example given in the readme.
I get this error
Neighborhood calculation time = 0.243874 secs.
Neighborhood calculation time = 0.249840 secs.
Neighborhood calculation time = 0.193178 secs.
Neighborhood calculation time = 0.205201 secs.
Neighborhood calculation time = 0.230796 secs.
Neighborhood calculation time = 0.317984 secs.
Neighborhood calculation time = 0.238225 secs.
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 22/22 [02:52<00:00, 7.83s/it]
ADD_passed 0.0
best_score 0.0
best check point saved in /home/robotlab/Desktop/HAOV/ZebraPose/6dof_pose_experiments/experiments/checkpoints/exp_lmo_BOPape/best_score/0_0000step0
exp_lmo_BOP iteration_step: 1 loss_b: 0.7003157009678205 loss_m: 0.4887707233428955 loss: 2.589717826246357
exp_lmo_BOP iteration_step: 2 loss_b: 0.69137206054109 loss_m: 0.3945087790489197 loss: 2.46862496067219
exp_lmo_BOP iteration_step: 3 loss_b: 0.694508611202345 loss_m: 0.31038543581962585 loss: 2.3939112694266607
exp_lmo_BOP iteration_step: 4 loss_b: 0.6951448561501329 loss_m: 0.2788199484348297 loss: 2.3642545168852283
exp_lmo_BOP iteration_step: 5 loss_b: 0.6913055153449651 loss_m: 0.22996273636817932 loss: 2.3038792824030745
exp_lmo_BOP iteration_step: 6 loss_b: 0.6898123773388913 loss_m: 0.25095099210739136 loss: 2.320388124124065
exp_lmo_BOP iteration_step: 7 loss_b: 0.6898932232178236 loss_m: 0.22556562721729279 loss: 2.295245296870764
exp_lmo_BOP iteration_step: 8 loss_b: 0.689580717264042 loss_m: 0.21235992014408112 loss: 2.281102071936207
exp_lmo_BOP iteration_step: 9 loss_b: 0.6850530220622502 loss_m: 0.20650124549865723 loss: 2.2616603116854077
exp_lmo_BOP iteration_step: 10 loss_b: 0.6861161840507828 loss_m: 0.19408346712589264 loss: 2.252432019278241
exp_lmo_BOP iteration_step: 11 loss_b: 0.685705219347128 loss_m: 0.19636832177639008 loss: 2.253483979817774
exp_lmo_BOP iteration_step: 12 loss_b: 0.6799169770450811 loss_m: 0.19241726398468018 loss: 2.2321681951199235
exp_lmo_BOP iteration_step: 13 loss_b: 0.6819430279221167 loss_m: 0.18748752772808075 loss: 2.233316611494431
exp_lmo_BOP iteration_step: 14 loss_b: 0.6840540645380042 loss_m: 0.18512609601020813 loss: 2.237288289624221
Traceback (most recent call last):
File "train.py", line 383, in
main(configs)
File "train.py", line 225, in main
for batch_idx, (data, entire_masks, masks, Rs, ts, Bboxes, class_code_images, cam_Ks) in enumerate(train_loader):
File "/home/robotlab/mambaforge/envs/zebrapose/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 521, in next
data = self._next_data()
File "/home/robotlab/mambaforge/envs/zebrapose/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1203, in _next_data
return self._process_data(data)
File "/home/robotlab/mambaforge/envs/zebrapose/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1229, in _process_data
data.reraise()
File "/home/robotlab/mambaforge/envs/zebrapose/lib/python3.8/site-packages/torch/_utils.py", line 434, in reraise
raise exception
TypeError: Caught TypeError in DataLoader worker process 1.
Original Traceback (most recent call last):
File "/home/robotlab/mambaforge/envs/zebrapose/lib/python3.8/site-packages/torch/utils/data/_utils/worker.py", line 287, in _worker_loop
data = fetcher.fetch(index)
File "/home/robotlab/mambaforge/envs/zebrapose/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py", line 49, in fetch
data = [self.dataset[idx] for idx in possibly_batched_index]
File "/home/robotlab/mambaforge/envs/zebrapose/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py", line 49, in
data = [self.dataset[idx] for idx in possibly_batched_index]
File "/home/robotlab/Desktop/HAOV/ZebraPose/zebrapose/bop_dataset_pytorch.py", line 263, in getitem
x = self.apply_augmentation(x)
File "/home/robotlab/Desktop/HAOV/ZebraPose/zebrapose/bop_dataset_pytorch.py", line 351, in apply_augmentation
x = augmentations.augment_image(x)
File "/home/robotlab/mambaforge/envs/zebrapose/lib/python3.8/site-packages/imgaug/augmenters/meta.py", line 763, in augment_image
"Expected to get a single numpy array of shape (H,W) or (H,W,C) "
TypeError: %d format: a number is required, not str
The config is the same as "exp_lmo_BOP.txt" but because i have a 2080Ti with 11GB of vram i have chosen my number of workers to 4 using a batch size of 16.
I believe my file structure is correct, but I'm not entirely sure. Here it is for lmo.
Any ideas as to what might cause this?
Hello, I am working on reproducing this project. I encountered the issue shown in the image while installing the dependencies for Progressive-X. I have searched the search engine for solutions to the related problem but couldn't resolve it. I would like to ask for guidance on how to address this issue. My OpenCV version is 4.3.0.36.
Hello,
I would like to ask if there is the JSON results of GDRNPPDet?
Dear author, can windows10 run your project?
the definition of ycbv_symmetry_obj in common_dataset_info.py is
ycbv_symmetry_obj = {
'bowl',
'wood_block',
'large_clamp',
'extra_large_clamp',
'foam_brick'
}
but large_marker is symmetry_obj in models_info.json, why it is not in ycbv_symmetry_obj?
I'm trying to use my own dataset, and i'm troubleshooting some training problems
I hacked together a way to visualize the input data as well as the masks, which works okay.
I noticed that the in train.py there is the following uncommented code.
#visulize_input_data_and_network(writer, train_loader, net)
Does anyone still have this code? The function doesn't exist in this repo
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.