vt-vl-lab / ican Goto Github PK
View Code? Open in Web Editor NEW[BMVC 2018] iCAN: Instance-Centric Attention Network for Human-Object Interaction Detection
Home Page: https://gaochen315.github.io/iCAN/
License: MIT License
[BMVC 2018] iCAN: Instance-Centric Attention Network for Human-Object Interaction Detection
Home Page: https://gaochen315.github.io/iCAN/
License: MIT License
hi,dear sir,have you recently made any new research on human-object-interaction?
I want to perform transfer learning to train the model with the pretrained weights on my own small image dataset. Let me know how to go about it.
Hello, thank you for your work.Your code only has places for iteration, but not for batchsize and epoch.So how are these two parameters set?
Thank you very much.
Thanx for ur excellent job!
I got a issue when i tried to run a training.
Please help me if u r willing to. @gaochen315
Traceback (most recent call last):
File "tools/Train_ResNet_HICO.py", line 21, in
from models.train_Solver_HICO import train_net
File "/home/ydm/project-HOI/iCAN/tools/../lib/models/train_Solver_HICO.py", line 12, in
from ult.ult import Get_Next_Instance_HO_Neg_HICO
File "/home/ydm/project-HOI/iCAN/tools/../lib/ult/ult.py", line 22, in
from config import cfg
ImportError: cannot import name cfg
If you suspect this is an IPython bug, please report it at:
https://github.com/ipython/ipython/issues
or send an email to the mailing list at [email protected]
You can print a more detailed traceback right now with "%tb", or use "%debug"
to interactively debug it.
Extra-detailed tracebacks for bug-reporting purposes can be enabled via:
%config Application.verbose_crash=True
How can i fix it?
Thanx again.
Hi there,
What your group have done is amazing! I'm trying to see the performance on the V-COCO dataset, however I wonder would there be an annotation marks the mapping between the category id and the actual meaning? some stuff like:
{"objs":
{"15": "bench",
"19": "horse",
"9": "boat"}
"verbs":
{"1": "sit",
"6": "play",
"9": "throw"}
}
I looked up the COCO official site, but only the object part is found whereas the list of verbs are missing.
I would appreciate it if you can help!
how do you test the hico-det and what's your metrics?
May i ask what's the meaning of the data in Trainval_Neg_VCOCO.pkl?
The data in Trainval_GT_VCOCO.pkl is the groundtruth of the bounding box,but i don't know what's the meaning of the data in Trainval_Neg_VCOCO.pkl
Hello,
First off, thank you for making the code public... I am excited to test the model. I followed the steps for the test demo in the repo but for some reason, I do not have the prior_mask.pkl file or the action_index.json file. Is there some way I can download these to make the demo work?
Thank you for your code.
I want to know how to generate the negative samples' action classes?
Hello, thank you for your work. I have a question about the code.On line 326 of iCAN_ResNet50_VCOCO. py, why is fc7_SH and fc7_SO sent into the classifier?According to the structure diagram in the paper, shouldn't fc9_SH and fc9_SO after feature fusion be sent into the classifier?
Good Evening,
There seems to be lot of rule based post processing being done after getting the model outputs. (Forcing prediction to be snowboarding if snowboard is present etc.). Can you report the numbers without such post processing being applied? Evaluating the model after removing post-processing is giving me very bad results, so I'm not sure if I'm doing it properly. I couldn't find any mention of such numbers in the BMVC paper.
Thank you!
Hello
Negative samples have wrong relationships. Therefore minimizing loss does not mean the predictions for these negative samples tend to the wrong relations?What is the reason for using negative examples?
File "E:/case/iCAN-master/tools/r-demo.py", line 27, in
Detection = pickle.load( open( cfg.ROOT_DIR + "/demo/HOI_Detection.pkl", "rb" ) )
FileNotFoundError: [Errno 2] No such file or directory: 'E:\case\iCAN-master/demo/HOI_Detection.pkl'
Hi,
Thanks for an interesting work. May I know whether the detection result for the default setting (without objects known) on HICO_DET test set is available to public? As the current released result for HICO_DET is for the known objects setting. Thanks!
Hi @gaochen315! Many thanks for the great work!
I have some questions about the detector in your model.
I noticed that in the testing process, the human-object pair proposals are given in Data/Test_Faster_RCNN_R-50-PFN_2x_HICO_DET.pkl
, are these proposals come from the outputs of Detectron or other codebases?
In the training process, you used a pretrained ResNet50-FPN from tf-faster-rcnn as the detector, why not just use the detection results in Data/Test_Faster_RCNN_R-50-PFN_2x_HICO_DET.pkl
?
Thank you very much for your attention.😊
when running the code, it reports that:
File "tools/Train_ResNet_VCOCO.py", line 20, in <module>
from ult.config import cfg
SyntaxError: future feature divisions is not defined (config.py, line 8)
And I update the from __future__ import divisions
of config.py
with:
from __future__ import division
and solve this error. it seems a typo error of lib/ult/config.py
.
Thanks for sharing your code with us! iCAN is really a fantastic job, I mean that~
May I ask is there any code or tool to visualize the detection results on HICO_DET? I see that there exists one tools/Visualization.ipynb to visualize the detection results on V-COCO. I have tried it and it works well.
Hope to hearing from you~ thank you very much!
Hi Chen,
In order to test on my own images, I follow steps in the Demo/Test on your own images section.
I encountered an issue when doing the step3(Detect all objects).
(sence-graph) nadoudou@nadoudou-ThinkStation-P310:~/iCAN$ python ../tf-faster-rcnn/tools/Object_Detector.py --img_dir demo/ --img_format png --Demo_RCNN demo/Object_Detection.pkl
2018-12-11 15:02:48.221898: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.1 instructions, but these are available on your machine and could speed up CPU computations.
2018-12-11 15:02:48.221920: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.2 instructions, but these are available on your machine and could speed up CPU computations.
2018-12-11 15:02:48.221925: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX instructions, but these are available on your machine and could speed up CPU computations.
2018-12-11 15:02:48.221930: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX2 instructions, but these are available on your machine and could speed up CPU computations.
2018-12-11 15:02:48.221934: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use FMA instructions, but these are available on your machine and could speed up CPU computations.
2018-12-11 15:02:49.798437: W tensorflow/core/framework/op_kernel.cc:1158] Not found: Key resnet_v1_50/rpn_conv/3x3/weights not found in checkpoint
2018-12-11 15:02:49.798742: W tensorflow/core/framework/op_kernel.cc:1158] Not found: Key resnet_v1_50/rpn_conv/3x3/biases not found in checkpoint
2018-12-11 15:02:49.799248: W tensorflow/core/framework/op_kernel.cc:1158] Not found: Key resnet_v1_50/bbox_pred/biases not found in checkpoint
2018-12-11 15:02:49.799520: W tensorflow/core/framework/op_kernel.cc:1158] Not found: Key resnet_v1_50/bbox_pred/weights not found in checkpoint
2018-12-11 15:02:49.799539: W tensorflow/core/framework/op_kernel.cc:1158] Not found: Key resnet_v1_50/rpn_cls_score/weights not found in checkpoint
2018-12-11 15:02:49.800069: W tensorflow/core/framework/op_kernel.cc:1158] Not found: Key resnet_v1_50/rpn_cls_score/biases not found in checkpoint
2018-12-11 15:02:49.800601: W tensorflow/core/framework/op_kernel.cc:1158] Not found: Key resnet_v1_50/rpn_bbox_pred/weights not found in checkpoint
2018-12-11 15:02:49.801182: W tensorflow/core/framework/op_kernel.cc:1158] Not found: Key resnet_v1_50/rpn_bbox_pred/biases not found in checkpoint
2018-12-11 15:02:49.804062: W tensorflow/core/framework/op_kernel.cc:1158] Not found: Key resnet_v1_50/cls_score/weights not found in checkpoint
2018-12-11 15:02:49.804685: W tensorflow/core/framework/op_kernel.cc:1158] Not found: Key resnet_v1_50/cls_score/biases not found in checkpoint
Traceback (most recent call last):
File "../tf-faster-rcnn/tools/Object_Detector.py", line 68, in <module>
saver.restore(sess, tfmodel)
File "/home/nadoudou/anaconda2/envs/sence-graph/lib/python2.7/site-packages/tensorflow/python/training/saver.py", line 1548, in restore
{self.saver_def.filename_tensor_name: save_path})
File "/home/nadoudou/anaconda2/envs/sence-graph/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 789, in run
run_metadata_ptr)
File "/home/nadoudou/anaconda2/envs/sence-graph/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 997, in _run
feed_dict_string, options, run_metadata)
File "/home/nadoudou/anaconda2/envs/sence-graph/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1132, in _do_run
target_list, options, run_metadata)
File "/home/nadoudou/anaconda2/envs/sence-graph/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1152, in _do_call
raise type(e)(node_def, op, message)
NotFoundError: Key resnet_v1_50/rpn_conv/3x3/weights not found in checkpoint
[[Node: save/RestoreV2_274 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_274/tensor_names, save/RestoreV2_274/shape_and_slices)]]
Caused by op u'save/RestoreV2_274', defined at:
File "../tf-faster-rcnn/tools/Object_Detector.py", line 67, in <module>
saver = tf.train.Saver()
File "/home/nadoudou/anaconda2/envs/sence-graph/lib/python2.7/site-packages/tensorflow/python/training/saver.py", line 1139, in __init__
self.build()
File "/home/nadoudou/anaconda2/envs/sence-graph/lib/python2.7/site-packages/tensorflow/python/training/saver.py", line 1170, in build
restore_sequentially=self._restore_sequentially)
File "/home/nadoudou/anaconda2/envs/sence-graph/lib/python2.7/site-packages/tensorflow/python/training/saver.py", line 691, in build
restore_sequentially, reshape)
File "/home/nadoudou/anaconda2/envs/sence-graph/lib/python2.7/site-packages/tensorflow/python/training/saver.py", line 407, in _AddRestoreOps
tensors = self.restore_op(filename_tensor, saveable, preferred_shard)
File "/home/nadoudou/anaconda2/envs/sence-graph/lib/python2.7/site-packages/tensorflow/python/training/saver.py", line 247, in restore_op
[spec.tensor.dtype])[0])
File "/home/nadoudou/anaconda2/envs/sence-graph/lib/python2.7/site-packages/tensorflow/python/ops/gen_io_ops.py", line 640, in restore_v2
dtypes=dtypes, name=name)
File "/home/nadoudou/anaconda2/envs/sence-graph/lib/python2.7/site-packages/tensorflow/python/framework/op_def_library.py", line 767, in apply_op
op_def=op_def)
File "/home/nadoudou/anaconda2/envs/sence-graph/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 2506, in create_op
original_op=self._default_original_op, op_def=op_def)
File "/home/nadoudou/anaconda2/envs/sence-graph/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 1269, in __init__
self._traceback = _extract_stack()
NotFoundError (see above for traceback): Key resnet_v1_50/rpn_conv/3x3/weights not found in checkpoint
[[Node: save/RestoreV2_274 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_274/tensor_names, save/RestoreV2_274/shape_and_slices)]]
many thanks!
Hi Chen,
When I trained an iCAN on V-COCO, the network read data from .pkl file which was downloaded from google Driver. Could you please tell me how to put the images dataset in a .pkl file? Do you have the code?
Thanks a lot.
Thanks for the awesome work.
I am trying to run and reproduce the results.
I find a problem: in Trainval_GT_VCOCO.pkl, there are 4971 distinct image ids with 8542 training instances.
While in vcoco_trainval.ids, there are actually 5400 distinct image ids.
My question is why image ids in Trainval_GT_VCOCO.pkl is different with Trainval_GT_VCOCO.pkl?
Is it because some people or objects cannot be detected in V-COCO images using FasterRCNN-FPN?
Hey @gaochen315, First of all thanks for sharing this great work. I am looking for executing this model directly on videos. Have you prepared script for that also ?
Hi, what do the rear three elements in each list in 'Trainval_Neg_HICO.pkl' mean?
for example,
[
2, # image_id
53, # action_HO, negative HOI
array([257.58002 , 2.112835, 631.65155 , 128.04715 ], dtype=float32), # Human bbox
array([129.54727 , 65.349594, 180.34726 , 148.78911 ], dtype=float32), # Object bbox
[46, 47, 48, 49, 50, 51, 52, 53], # 1
40, # 2
array(0.7580195, dtype=float32) # 3
]
can you illustrate the last three elements?
By the way, do you change the order of 80 objects in COCO(or VCOCO)?
Thanks for the repo. In the supplemental material, you guys said to augment data by spatial jittering in the ground truth. Can you give a bit more details on this? Like how do you generate the extra ground truth samples? Also, you mentioned the human and object stream losses are calculated on the 16 positive triplets, what does mean by that?
Hi,
I would like to know the detector performance on HICO-Det dataset. Could you report the AP?
On HICO-DET dataset, the same object may be annotated by several bouding-boxes which have high IOU with each other. I wonder if you have done something for it, when training the detector.
Hello,
V-COCO dataset has two ways of calculating Role mAP. May I know which one did you guys report on the paper?
Hi, would you like to share an available link for '/Weights/res50_faster_rcnn_iter_1190000.ckpt' this pretained model? Many thanks!
Hello
You haven't put the file "HOI_iter_1800000.ckpt"somewhere?I wish to perform testing without training on cpu?
Hi @gaochen315
Sorry to bother. I have a quick question that maybe you can help.
I wonder how can i generate my own "Trainval_GT_VCOCO.pkl"and "Trainval_Neg_VCOCO.pkl"?
To my understanding, i can install Faster R-CNN and run it on test set of VCOCO to get the test data (to obtain "Test_Faster_RCNN_R-50-PFN_2x_VCOCO.pkl") .
But how can i generate the training data?
Thanx a lot.
Hi, can thIs code run with python3.7 and tensorflow1.13 version ?
Hi @gaochen315, thanks for the great work.
I have a question about the test time. If we assume that there are m persons and n objects, so the test_net need to process N=m*(m+n-1) times, and N might be very large, how much time do you use when testing?
Since I have not run your code yet, but when I do it on my own with pytorch it will cost much time.
Thanks a lot!
Hello again,
Would it be possible to use the iCAN model for real-time HOI output? i.e. on a live video stream (in online mode); if yes, then could you guide me on the steps required to achieve this?
Thank you,
Hi,
Thanks for the awesome work that you are doing. I am really impressed with the way the model works out. However, when testing the custom images steps, I did all the steps correctly. Also, I guess both the object detection and HOI detection commands were successful without any errors. However, when trying to demo the Djokovic image I get no bounding boxes and no HOI detections.
There was no prior_mask.pkl and action_index.json files. I downloaded them from your drive and placed them in the Data folder in order to run the model on the Djokovic images.
I guess the error is because of prior_mask.pkl file? But I'm not sure. Can you help out?
Hello,
Thank you for releasing the code. On the home page, you have a demo on a video (two people talking from the Big Bang Theory). In the README, you have instructions for Demo/Test on our own images.
Could you guide me on the process to do a demo on my own video? what are the steps to obtaining the results on an mp4 file?
Thank you,
Hi,
I would like to train the model on HICO-Det dataset.
I trained my model on a single TITAN X GPU for HICO-DET dataset. It took about five days for iteration 1800000 times.It takes too long.
would it be possible to train the model on multiple GPUs? what changes would I have to make?
and How big is the batch_size of your network?
Thanks!
Hi, I tested the demo.ipynb, the visualization of your demo image "Djokovic_000X" runs good, while then I put my own picture in the 'demo/' folder, the visualization is not working. There is a line in the code saying "if (ele['image_id'] == image_id):", it seems if picture id is not in ele[], it will not work. Then I changed the "Djokovic_0005.png" into "55.png", there are no more boxes showing in the picture, just as my picture did. After I commenting this line, the boxes always show the same contents as visualizing "Djokovic_000X", no matter how I change the input picture.
Is there anything I did wrong in my steps of visualizing own pictures?
HI ,
I try to run "Download_data.py" for some resources,but always show:
" Max retries exceeded with url: /uc?export=download&id=1IbR4kiWgLF8seaKjOMmwaHs0Bfwl5Dq1 (Caused by NewConnectionError('<requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x6ffff81b990>: Failed to establish a new connection: [Errno 116] Connection timed out',))",but i can open "https://docs.google.com",excuse for this URL is right?
The paper described that the object instances were detected by Faster R-CNN, in this codes, however, did you just input the ground truth boxes directly?
Hi, what do the elements in 'Reporting Error Analysis' mean?
for example:
[ hold-obj: (inc)68.07 (bck)0.00 (H_mis)0.87 (O_mis)3.19 (mis-gr)14.21 occl(13.66)
(rec:29.60 = #tp:1068/#pos:3608) (prec:29.60 = #tp:1068/#det:4044)
]
can you illustrate the elements?
inc:
bck:
H_mis:
O_mis
mis-gr:
occl:
prec:
det:
Thank you,
Thanks for sharing your code. @gaochen315
I wonder how can i successfully visualize the attention maps like those in your paper?
Could anyone help? Thanx a lot.
Hello, this code is very helpful. I did run the training script and I found that the model is saved with checkpoints every 20000 iterations. But, when the training if over how to use the final model in the code. I see that you use a .pkl file in Results/300000_iCAN_ResNet50_VCOCO.pkl so how I use the the new trained model.
In addition to, I need to know what do you mean by the files
Trainval_GT
Trainval_N
or what are there content?
Thank you.
you mention that you developed the model using CUDA 8.0.
I have a few questions about GPU training:
how many GPUs did you use to train the model? (for e.g python tools/Train_ResNet_VCOCO.py --model iCAN_ResNet50_VCOCO --num_iteration 300000)
which type of GPU?
approx how long did the training take?
would it be possible to train the model on multiple GPUs? what changes would I have to make?
thank you,
iCAN is an interesting work.
In this paper, you said, "Additional results including detailed class-wise performance and error diagnosis can be found in the supplementary material".
Could you release this supplemental?
I want to detect a stealing action, such as stealing a shoe(a person put this shoe in his packet or handbag)。How many videos should I have to train ? What is the procedure to train ?
Hello, thank you for your work.When I was testing the HICO dataset, I generated a lot of MAT files. How can I output the results?
For generating HICO detections, I couldn't able to locate the file lib/ult/Generate_HICO_detection file is missing.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.