vt-vl-lab / ican Goto Github PK

Hello,

First off, thank you for making the code public... I am excited to test the model. I followed the steps for the test demo in the repo but for some reason, I do not have the prior_mask.pkl file or the action_index.json file. Is there some way I can download these to make the demo work?

you mention that you developed the model using CUDA 8.0.

I have a few questions about GPU training:

how many GPUs did you use to train the model? (for e.g python tools/Train_ResNet_VCOCO.py --model iCAN_ResNet50_VCOCO --num_iteration 300000)

which type of GPU?

approx how long did the training take?

would it be possible to train the model on multiple GPUs? what changes would I have to make?

thank you,

Good Evening,

There seems to be lot of rule based post processing being done after getting the model outputs. (Forcing prediction to be snowboarding if snowboard is present etc.). Can you report the numbers without such post processing being applied? Evaluating the model after removing post-processing is giving me very bad results, so I'm not sure if I'm doing it properly. I couldn't find any mention of such numbers in the BMVC paper.

Thank you!

Thanks for the awesome work.
I am trying to run and reproduce the results.

I find a problem: in Trainval_GT_VCOCO.pkl, there are 4971 distinct image ids with 8542 training instances.
While in vcoco_trainval.ids, there are actually 5400 distinct image ids.

My question is why image ids in Trainval_GT_VCOCO.pkl is different with Trainval_GT_VCOCO.pkl?
Is it because some people or objects cannot be detected in V-COCO images using FasterRCNN-FPN?

Hello, thank you for your work.Your code only has places for iteration, but not for batchsize and epoch.So how are these two parameters set?
Thank you very much.

May i ask what's the meaning of the data in Trainval_Neg_VCOCO.pkl?
The data in Trainval_GT_VCOCO.pkl is the groundtruth of the bounding box,but i don't know what's the meaning of the data in Trainval_Neg_VCOCO.pkl

Hi Chen:
While I test on my own images as you said, I could not find the pkl files, both HOI_Detection.pkl and Object_Detection.pkl , where can I get its?
many thanks.
Edward

File "E:/case/iCAN-master/tools/r-demo.py", line 27, in
Detection = pickle.load( open( cfg.ROOT_DIR + "/demo/HOI_Detection.pkl", "rb" ) )
FileNotFoundError: [Errno 2] No such file or directory: 'E:\case\iCAN-master/demo/HOI_Detection.pkl'

Hi @gaochen315, thanks for the great work.

I have a question about the test time. If we assume that there are m persons and n objects, so the test_net need to process N=m*(m+n-1) times, and N might be very large, how much time do you use when testing?

Since I have not run your code yet, but when I do it on my own with pytorch it will cost much time.

Thanks a lot!

when running the code, it reports that:

 File "tools/Train_ResNet_VCOCO.py", line 20, in <module>
    from ult.config import cfg
SyntaxError: future feature divisions is not defined (config.py, line 8)

And I update the from __future__ import divisions of config.py with:
from __future__ import division

and solve this error. it seems a typo error of lib/ult/config.py.

Hi,
Thanks for the awesome work that you are doing. I am really impressed with the way the model works out. However, when testing the custom images steps, I did all the steps correctly. Also, I guess both the object detection and HOI detection commands were successful without any errors. However, when trying to demo the Djokovic image I get no bounding boxes and no HOI detections.

There was no prior_mask.pkl and action_index.json files. I downloaded them from your drive and placed them in the Data folder in order to run the model on the Djokovic images.

I guess the error is because of prior_mask.pkl file? But I'm not sure. Can you help out?

Hi there,

What your group have done is amazing! I'm trying to see the performance on the V-COCO dataset, however I wonder would there be an annotation marks the mapping between the category id and the actual meaning? some stuff like:
{"objs":
{"15": "bench",
"19": "horse",
"9": "boat"}
"verbs":
{"1": "sit",
"6": "play",
"9": "throw"}
}

I looked up the COCO official site, but only the object part is found whereas the list of verbs are missing.

I would appreciate it if you can help!

Hi, I tested the demo.ipynb, the visualization of your demo image "Djokovic_000X" runs good, while then I put my own picture in the 'demo/' folder, the visualization is not working. There is a line in the code saying "if (ele['image_id'] == image_id):", it seems if picture id is not in ele[], it will not work. Then I changed the "Djokovic_0005.png" into "55.png", there are no more boxes showing in the picture, just as my picture did. After I commenting this line, the boxes always show the same contents as visualizing "Djokovic_000X", no matter how I change the input picture.

Is there anything I did wrong in my steps of visualizing own pictures?

Hi, would you like to share an available link for '/Weights/res50_faster_rcnn_iter_1190000.ckpt' this pretained model? Many thanks!

Hi，
I would like to know the detector performance on HICO-Det dataset. Could you report the AP?
On HICO-DET dataset, the same object may be annotated by several bouding-boxes which have high IOU with each other. I wonder if you have done something for it, when training the detector.

Hi @gaochen315
Sorry to bother. I have a quick question that maybe you can help.
I wonder how can i generate my own "Trainval_GT_VCOCO.pkl"and "Trainval_Neg_VCOCO.pkl"?
To my understanding, i can install Faster R-CNN and run it on test set of VCOCO to get the test data (to obtain "Test_Faster_RCNN_R-50-PFN_2x_VCOCO.pkl") .
But how can i generate the training data?

Thanx a lot.

Hi，
I would like to train the model on HICO-Det dataset.

I trained my model on a single TITAN X GPU for HICO-DET dataset. It took about five days for iteration 1800000 times.It takes too long.

would it be possible to train the model on multiple GPUs? what changes would I have to make?

and How big is the batch_size of your network?

Thanks!

Hello, thank you for your work.When I was testing the HICO dataset, I generated a lot of MAT files. How can I output the results？

Hello, this code is very helpful. I did run the training script and I found that the model is saved with checkpoints every 20000 iterations. But, when the training if over how to use the final model in the code. I see that you use a .pkl file in Results/300000_iCAN_ResNet50_VCOCO.pkl so how I use the the new trained model.

In addition to, I need to know what do you mean by the files
Trainval_GT
Trainval_N
or what are there content?

Thank you.

Hello,
V-COCO dataset has two ways of calculating Role mAP. May I know which one did you guys report on the paper?

I want to perform transfer learning to train the model with the pretrained weights on my own small image dataset. Let me know how to go about it.

Hey @gaochen315, First of all thanks for sharing this great work. I am looking for executing this model directly on videos. Have you prepared script for that also ?

Hi, what do the elements in 'Reporting Error Analysis' mean?
for example:
[ hold-obj: (inc)68.07 (bck)0.00 (H_mis)0.87 (O_mis)3.19 (mis-gr)14.21 occl(13.66)
(rec:29.60 = #tp:1068/#pos:3608) (prec:29.60 = #tp:1068/#det:4044)
]
can you illustrate the elements?
inc:
bck:
H_mis:
O_mis
mis-gr:
occl:
prec:
det:

Thank you,

Thanx for ur excellent job!
I got a issue when i tried to run a training.
Please help me if u r willing to. @gaochen315

I tired to run: python tools/Train_ResNet_HICO.py --num_iteration 1800000 but got this information:

Traceback (most recent call last):
File "tools/Train_ResNet_HICO.py", line 21, in
from models.train_Solver_HICO import train_net
File "/home/ydm/project-HOI/iCAN/tools/../lib/models/train_Solver_HICO.py", line 12, in
from ult.ult import Get_Next_Instance_HO_Neg_HICO
File "/home/ydm/project-HOI/iCAN/tools/../lib/ult/ult.py", line 22, in
from config import cfg
ImportError: cannot import name cfg

If you suspect this is an IPython bug, please report it at:
https://github.com/ipython/ipython/issues
or send an email to the mailing list at [email protected]

You can print a more detailed traceback right now with "%tb", or use "%debug"
to interactively debug it.

Extra-detailed tracebacks for bug-reporting purposes can be enabled via:
%config Application.verbose_crash=True

I set my environment as u said and i tried tensorflow 1.1 and 1.2

How can i fix it?
Thanx again.

Hi, can thIs code run with python3.7 and tensorflow1.13 version ?

Hi,
Thanks for an interesting work. May I know whether the detection result for the default setting (without objects known) on HICO_DET test set is available to public? As the current released result for HICO_DET is for the known objects setting. Thanks!

Hello
Negative samples have wrong relationships. Therefore minimizing loss does not mean the predictions for these negative samples tend to the wrong relations?What is the reason for using negative examples?

Hi GaoChen,

#35
I have the same problem, can you give me some detailed tips

Thanks,
Tairan Chen

Hello again,

Would it be possible to use the iCAN model for real-time HOI output? i.e. on a live video stream (in online mode); if yes, then could you guide me on the steps required to achieve this?

Thank you,

For generating HICO detections, I couldn't able to locate the file lib/ult/Generate_HICO_detection file is missing.

Hello,

Thank you for releasing the code. On the home page, you have a demo on a video (two people talking from the Big Bang Theory). In the README, you have instructions for Demo/Test on our own images.

Could you guide me on the process to do a demo on my own video? what are the steps to obtaining the results on an mp4 file?

Thank you,

how do you test the hico-det and what's your metrics?

iCAN is an interesting work.
In this paper, you said, "Additional results including detailed class-wise performance and error diagnosis can be found in the supplementary material".

Could you release this supplemental?

Thanks for sharing your code. @gaochen315
I wonder how can i successfully visualize the attention maps like those in your paper?
Could anyone help? Thanx a lot.

The paper described that the object instances were detected by Faster R-CNN, in this codes, however, did you just input the ground truth boxes directly?

hi
thanks for your code sharing
and when I try to train a model used VCOCO datasets, I was doubt why the loss is so small from the beginning, like this:

I want to know if I make some mistake to do the training step or the loss is same as yours?
thanks again~

Thank you for your code.
I want to know how to generate the negative samples' action classes?

Thanks for the repo. In the supplemental material, you guys said to augment data by spatial jittering in the ground truth. Can you give a bit more details on this? Like how do you generate the extra ground truth samples? Also, you mentioned the human and object stream losses are calculated on the 16 positive triplets, what does mean by that?

Hi @gaochen315! Many thanks for the great work!

I have some questions about the detector in your model.

1. I noticed that in the testing process, the human-object pair proposals are given in Data/Test_Faster_RCNN_R-50-PFN_2x_HICO_DET.pkl, are these proposals come from the outputs of Detectron or other codebases?

2. In the training process, you used a pretrained ResNet50-FPN from tf-faster-rcnn as the detector, why not just use the detection results in Data/Test_Faster_RCNN_R-50-PFN_2x_HICO_DET.pkl?

Thank you very much for your attention.😊

Hi, what do the rear three elements in each list in 'Trainval_Neg_HICO.pkl' mean?
for example,
[
2, # image_id
53, # action_HO, negative HOI
array([257.58002 , 2.112835, 631.65155 , 128.04715 ], dtype=float32), # Human bbox
array([129.54727 , 65.349594, 180.34726 , 148.78911 ], dtype=float32), # Object bbox
[46, 47, 48, 49, 50, 51, 52, 53], # 1
40, # 2
array(0.7580195, dtype=float32) # 3
]
can you illustrate the last three elements?

By the way, do you change the order of 80 objects in COCO(or VCOCO)?

Hi Chen,
When I trained an iCAN on V-COCO, the network read data from .pkl file which was downloaded from google Driver. Could you please tell me how to put the images dataset in a .pkl file? Do you have the code?

Thanks a lot.

Thanks for sharing your code with us! iCAN is really a fantastic job, I mean that~
May I ask is there any code or tool to visualize the detection results on HICO_DET? I see that there exists one tools/Visualization.ipynb to visualize the detection results on V-COCO. I have tried it and it works well.
Hope to hearing from you~ thank you very much!

I want to detect a stealing action, such as stealing a shoe(a person put this shoe in his packet or handbag)。How many videos should I have to train ? What is the procedure to train ?

Hello, thank you for your work. I have a question about the code.On line 326 of iCAN_ResNet50_VCOCO. py, why is fc7_SH and fc7_SO sent into the classifier?According to the structure diagram in the paper, shouldn't fc9_SH and fc9_SO after feature fusion be sent into the classifier?

i used my own dataset(hico style) to train the model, There are six categories， However, during testing, the output of some categories was greater than 1。

Hello
You haven't put the file "HOI_iter_1800000.ckpt"somewhere?I wish to perform testing without training on cpu?

Hi Chen,
In order to test on my own images, I follow steps in the Demo/Test on your own images section.
I encountered an issue when doing the step3(Detect all objects).

(sence-graph) nadoudou@nadoudou-ThinkStation-P310:~/iCAN$ python ../tf-faster-rcnn/tools/Object_Detector.py --img_dir demo/ --img_format png --Demo_RCNN demo/Object_Detection.pkl
2018-12-11 15:02:48.221898: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.1 instructions, but these are available on your machine and could speed up CPU computations.
2018-12-11 15:02:48.221920: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.2 instructions, but these are available on your machine and could speed up CPU computations.
2018-12-11 15:02:48.221925: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX instructions, but these are available on your machine and could speed up CPU computations.
2018-12-11 15:02:48.221930: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX2 instructions, but these are available on your machine and could speed up CPU computations.
2018-12-11 15:02:48.221934: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use FMA instructions, but these are available on your machine and could speed up CPU computations.
2018-12-11 15:02:49.798437: W tensorflow/core/framework/op_kernel.cc:1158] Not found: Key resnet_v1_50/rpn_conv/3x3/weights not found in checkpoint
2018-12-11 15:02:49.798742: W tensorflow/core/framework/op_kernel.cc:1158] Not found: Key resnet_v1_50/rpn_conv/3x3/biases not found in checkpoint
2018-12-11 15:02:49.799248: W tensorflow/core/framework/op_kernel.cc:1158] Not found: Key resnet_v1_50/bbox_pred/biases not found in checkpoint
2018-12-11 15:02:49.799520: W tensorflow/core/framework/op_kernel.cc:1158] Not found: Key resnet_v1_50/bbox_pred/weights not found in checkpoint
2018-12-11 15:02:49.799539: W tensorflow/core/framework/op_kernel.cc:1158] Not found: Key resnet_v1_50/rpn_cls_score/weights not found in checkpoint
2018-12-11 15:02:49.800069: W tensorflow/core/framework/op_kernel.cc:1158] Not found: Key resnet_v1_50/rpn_cls_score/biases not found in checkpoint
2018-12-11 15:02:49.800601: W tensorflow/core/framework/op_kernel.cc:1158] Not found: Key resnet_v1_50/rpn_bbox_pred/weights not found in checkpoint
2018-12-11 15:02:49.801182: W tensorflow/core/framework/op_kernel.cc:1158] Not found: Key resnet_v1_50/rpn_bbox_pred/biases not found in checkpoint
2018-12-11 15:02:49.804062: W tensorflow/core/framework/op_kernel.cc:1158] Not found: Key resnet_v1_50/cls_score/weights not found in checkpoint
2018-12-11 15:02:49.804685: W tensorflow/core/framework/op_kernel.cc:1158] Not found: Key resnet_v1_50/cls_score/biases not found in checkpoint
Traceback (most recent call last):
  File "../tf-faster-rcnn/tools/Object_Detector.py", line 68, in <module>
    saver.restore(sess, tfmodel)
  File "/home/nadoudou/anaconda2/envs/sence-graph/lib/python2.7/site-packages/tensorflow/python/training/saver.py", line 1548, in restore
    {self.saver_def.filename_tensor_name: save_path})
  File "/home/nadoudou/anaconda2/envs/sence-graph/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 789, in run
    run_metadata_ptr)
  File "/home/nadoudou/anaconda2/envs/sence-graph/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 997, in _run
    feed_dict_string, options, run_metadata)
  File "/home/nadoudou/anaconda2/envs/sence-graph/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1132, in _do_run
    target_list, options, run_metadata)
  File "/home/nadoudou/anaconda2/envs/sence-graph/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1152, in _do_call
    raise type(e)(node_def, op, message)
NotFoundError: Key resnet_v1_50/rpn_conv/3x3/weights not found in checkpoint
	 [[Node: save/RestoreV2_274 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_274/tensor_names, save/RestoreV2_274/shape_and_slices)]]

Caused by op u'save/RestoreV2_274', defined at:
  File "../tf-faster-rcnn/tools/Object_Detector.py", line 67, in <module>
    saver = tf.train.Saver()
  File "/home/nadoudou/anaconda2/envs/sence-graph/lib/python2.7/site-packages/tensorflow/python/training/saver.py", line 1139, in __init__
    self.build()
  File "/home/nadoudou/anaconda2/envs/sence-graph/lib/python2.7/site-packages/tensorflow/python/training/saver.py", line 1170, in build
    restore_sequentially=self._restore_sequentially)
  File "/home/nadoudou/anaconda2/envs/sence-graph/lib/python2.7/site-packages/tensorflow/python/training/saver.py", line 691, in build
    restore_sequentially, reshape)
  File "/home/nadoudou/anaconda2/envs/sence-graph/lib/python2.7/site-packages/tensorflow/python/training/saver.py", line 407, in _AddRestoreOps
    tensors = self.restore_op(filename_tensor, saveable, preferred_shard)
  File "/home/nadoudou/anaconda2/envs/sence-graph/lib/python2.7/site-packages/tensorflow/python/training/saver.py", line 247, in restore_op
    [spec.tensor.dtype])[0])
  File "/home/nadoudou/anaconda2/envs/sence-graph/lib/python2.7/site-packages/tensorflow/python/ops/gen_io_ops.py", line 640, in restore_v2
    dtypes=dtypes, name=name)
  File "/home/nadoudou/anaconda2/envs/sence-graph/lib/python2.7/site-packages/tensorflow/python/framework/op_def_library.py", line 767, in apply_op
    op_def=op_def)
  File "/home/nadoudou/anaconda2/envs/sence-graph/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 2506, in create_op
    original_op=self._default_original_op, op_def=op_def)
  File "/home/nadoudou/anaconda2/envs/sence-graph/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 1269, in __init__
    self._traceback = _extract_stack()

NotFoundError (see above for traceback): Key resnet_v1_50/rpn_conv/3x3/weights not found in checkpoint
	 [[Node: save/RestoreV2_274 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_274/tensor_names, save/RestoreV2_274/shape_and_slices)]]

many thanks!

Thanks for your awesome work! I have a question.
example data in 'Trainval_GT_VCOCO.pkl':
[210751, #image id
[25, 13], #2
array([272.8 , 102.62, 489.02, 371.2 ]), #human box
array([421.44, 158.25, 478.95, 202.4 ]), #object box
[25, 10, 13]] #5

What does #2 and #5 mean? Thanks a lot.

HI ,
I try to run "Download_data.py" for some resources,but always show:
" Max retries exceeded with url: /uc?export=download&id=1IbR4kiWgLF8seaKjOMmwaHs0Bfwl5Dq1 (Caused by NewConnectionError('<requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x6ffff81b990>: Failed to establish a new connection: [Errno 116] Connection timed out',))",but i can open "https://docs.google.com",excuse for this URL is right?

hi，dear sir,have you recently made any new research on human-object-interaction?