kexinyi / ns-vqa Goto Github PK
View Code? Open in Web Editor NEWNeural-symbolic visual question answering
Neural-symbolic visual question answering
Hi, is the Mask RCNN only trained on the CLEVR or used with the original (official) pre-trained parameters?
May i ask,which part of your code mainly embodies symbol inference? Thx !!!
First of all, thanks for this code!
I am trying to adapt your code in order to have it take one of the images, a custom question, and intercept the answer given by the executor. For such questions, however, the only information that is given for each question is the image_index
and the question
(though I can artificially add a split
). I however do not seem to manage to modify to code to make it work without an input program associated to the questions.
Would you have any pointers in how I could go about doing this?
Hey,
Thanks for your work.
I have read your paper and want to redo it. But I find that my computer is lack of NVIDIA GPU.
Is it possible to run this project without a NVIDIA GPU?
Hey,
Thanks you for this work. Really enjoyed your paper.
Is it possible to share your detections.pkl with the masks for CLEVR? (or only the masks). Generating the masks using blender is too long.
Thanks.
Hello,
In the paper, you say: "Because the original CLEVR dataset does not include object masks, we generate these 4,000 training images ourselves using the CLEVR dataset generation tool". However, I don't see this process in this code release.
Can you release the code for how to do this, or give some insight into the process you used to do this?
Thanks!
Hi,
While compiling the libraries for mark R-CNN I got the following error:
gcc: error: /home/<username>/<path>/ns-vqa/scene_parse/mask_rcnn/lib/model/nms/src/nms_cuda.c: No such file or directory
Can you suggest any solution?
Thanks
python tools/test_net.py
--dataset clevr_original_val
--cfg configs/baselines/e2e_mask_rcnn_R-50-FPN_1x.yaml
--load_ckpt ../../data/pretrained/object_detector.pt
--output_dir ../../data/mask_rcnn/results/clevr_val_pretrained
When I run the script above, it gives out a cudaCheckError
cudaCheckError() failed : no kernel image is available for execution on the device .
it happens with the RoIAlignFunction
I was wondering if there are any updated links for downloading the smaller subset of CLEVR(~4k Images)? The current download.sh seems to be out of date.
Hi, I retrain the model but get lower accuracy (overall_acc=0.72 for pretrained parser and 0.88 for reinforce-finetuning), do you have any suggestions?
I'm trying to run pretrained models. I'm in Step 1: object detection.
I encounter this error:
Traceback (most recent call last):
File "/home/therese/PycharmProjects/neural_symbolic_vqa/ns-vqa/scene_parse/mask_rcnn/tools/test_net.py", line 128, in
check_expected_results=True)
File "/home/therese/PycharmProjects/neural_symbolic_vqa/ns-vqa/scene_parse/mask_rcnn/lib/core/test_engine.py", line 150, in run_inference
all_results = result_getter()
File "/home/therese/PycharmProjects/neural_symbolic_vqa/ns-vqa/scene_parse/mask_rcnn/lib/core/test_engine.py", line 130, in result_getter
multi_gpu=multi_gpu_testing
File "/home/therese/PycharmProjects/neural_symbolic_vqa/ns-vqa/scene_parse/mask_rcnn/lib/core/test_engine.py", line 180, in test_net_on_dataset
args, dataset_name, proposal_file, output_dir, gpu_id=gpu_id
File "/home/therese/PycharmProjects/neural_symbolic_vqa/ns-vqa/scene_parse/mask_rcnn/lib/core/test_engine.py", line 275, in test_net
cls_boxes_i, cls_segms_i, cls_keyps_i = im_detect_all(model, im, box_proposals, timers)
File "/home/therese/PycharmProjects/neural_symbolic_vqa/ns-vqa/scene_parse/mask_rcnn/lib/core/test.py", line 71, in im_detect_all
model, im, cfg.TEST.SCALE, cfg.TEST.MAX_SIZE, box_proposals)
File "/home/therese/PycharmProjects/neural_symbolic_vqa/ns-vqa/scene_parse/mask_rcnn/lib/core/test.py", line 152, in im_detect_bbox
return_dict = model(**inputs)
File "/home/therese/anaconda2/envs/ns-vqa/lib/python3.6/site-packages/torch/nn/modules/module.py", line 491, in call
result = self.forward(*input, **kwargs)
File "/home/therese/PycharmProjects/neural_symbolic_vqa/ns-vqa/scene_parse/mask_rcnn/lib/nn/parallel/data_parallel.py", line 108, in forward
outputs = [self.module(*inputs[0], **kwargs[0])]
File "/home/therese/anaconda2/envs/ns-vqa/lib/python3.6/site-packages/torch/nn/modules/module.py", line 491, in call
result = self.forward(*input, **kwargs)
File "/home/therese/PycharmProjects/neural_symbolic_vqa/ns-vqa/scene_parse/mask_rcnn/lib/modeling/model_builder.py", line 157, in forward
return self._forward(data, im_info, roidb, **rpn_kwargs)
File "/home/therese/PycharmProjects/neural_symbolic_vqa/ns-vqa/scene_parse/mask_rcnn/lib/modeling/model_builder.py", line 168, in _forward
blob_conv = self.Conv_Body(im_data)
File "/home/therese/anaconda2/envs/ns-vqa/lib/python3.6/site-packages/torch/nn/modules/module.py", line 491, in call
result = self.forward(*input, **kwargs)
File "/home/therese/PycharmProjects/neural_symbolic_vqa/ns-vqa/scene_parse/mask_rcnn/lib/modeling/FPN.py", line 246, in forward
self.topdown_lateral_modules[i](fpn_inner_blobs[-1], conv_body_blobs[-(i+2)])
File "/home/therese/anaconda2/envs/ns-vqa/lib/python3.6/site-packages/torch/nn/modules/module.py", line 491, in call
result = self.forward(*input, **kwargs)
File "/home/therese/PycharmProjects/neural_symbolic_vqa/ns-vqa/scene_parse/mask_rcnn/lib/modeling/FPN.py", line 310, in forward
return lat + td
RuntimeError: The size of tensor a (75) must match the size of tensor b (76) at non-singleton dimension 3
Anyone has encountered this before?
My pytorch version is 0.4.0
I get this error :
File "tools/preprocess_questions.py", line 11, in
import utils.programs as program_utils
ModuleNotFoundError: No module named 'utils'
Hi,
I encounter this error message
module 'modeling.roi_xfrom.roi_align._ext.roi_align' has no attribute 'roi_align_forward_cuda'
both running pretrain scene parser and training scene parser.
Do you have solution of this?
Thanks.
My envs setting is below:
python=3.5
pytorch=0.4.0
CUDA=9.0
when i run :
python tools/test_net.py
--dataset clevr_original_val
--cfg configs/baselines/e2e_mask_rcnn_R-50-FPN_1x.yaml
--load_ckpt ../../data/pretrained/object_detector.pt
--output_dir ../../data/mask_rcnn/results/clevr_val_pretrained
i get this error ImportError: cannot import name 'numpy_type_map'
i use CUDA10.1 pytorch 1.3 torchvision 0.4.1
Do you have the code for testing on CLEVR-Human?
Or can you briefly describe where to change for testing on CLEVR-Human.
Thank you.
I installed the environment requirements following every step as README.md said, but CUDNN error occured at the Step 1:object detection.
with the cmd:
python tools/train_net_step.py
--dataset clevr-mini
--cfg configs/baselines/e2e_mask_rcnn_R-50-FPN_1x.yaml
--bs 8
--set OUTPUT_DIR ../../data/mask_rcnn/outputs
Here is the Detail Error:
INFO test_engine.py: 331: loading checkpoint ../../data/pretrained/object_detector.pt
Traceback (most recent call last):
File "tools/test_net.py", line 126, in <module>
check_expected_results=True)
File "/home/tang/ns-vqa-master/scene_parse/mask_rcnn/lib/core/test_engine.py", line 129, in run_inference
all_results = result_getter()
File "/home/tang/ns-vqa-master/scene_parse/mask_rcnn/lib/core/test_engine.py", line 109, in result_getter
multi_gpu=multi_gpu_testing
File "/home/tang/ns-vqa-master/scene_parse/mask_rcnn/lib/core/test_engine.py", line 159, in test_net_on_dataset
args, dataset_name, proposal_file, output_dir, gpu_id=gpu_id
File "/home/tang/ns-vqa-master/scene_parse/mask_rcnn/lib/core/test_engine.py", line 254, in test_net
cls_boxes_i, cls_segms_i, cls_keyps_i = im_detect_all(model, im, box_proposals, timers)
File "/home/tang/ns-vqa-master/scene_parse/mask_rcnn/lib/core/test.py", line 71, in im_detect_all
model, im, cfg.TEST.SCALE, cfg.TEST.MAX_SIZE, box_proposals)
File "/home/tang/ns-vqa-master/scene_parse/mask_rcnn/lib/core/test.py", line 152, in im_detect_bbox
return_dict = model(**inputs)
File "/home/tang/anaconda3/envs/ns-vqa/lib/python3.6/site-packages/torch/nn/modules/module.py", line 491, in __call__
result = self.forward(*input, **kwargs)
File "/home/tang/ns-vqa-master/scene_parse/mask_rcnn/lib/nn/parallel/data_parallel.py", line 108, in forward
outputs = [self.module(*inputs[0], **kwargs[0])]
File "/home/tang/anaconda3/envs/ns-vqa/lib/python3.6/site-packages/torch/nn/modules/module.py", line 491, in __call__
result = self.forward(*input, **kwargs)
File "/home/tang/ns-vqa-master/scene_parse/mask_rcnn/lib/modeling/model_builder.py", line 144, in forward
return self._forward(data, im_info, roidb, **rpn_kwargs)
File "/home/tang/ns-vqa-master/scene_parse/mask_rcnn/lib/modeling/model_builder.py", line 155, in _forward
blob_conv = self.Conv_Body(im_data)
File "/home/tang/anaconda3/envs/ns-vqa/lib/python3.6/site-packages/torch/nn/modules/module.py", line 491, in __call__
result = self.forward(*input, **kwargs)
File "/home/tang/ns-vqa-master/scene_parse/mask_rcnn/lib/modeling/FPN.py", line 228, in forward
conv_body_blobs = [self.conv_body.res1(x)]
File "/home/tang/anaconda3/envs/ns-vqa/lib/python3.6/site-packages/torch/nn/modules/module.py", line 491, in __call__
result = self.forward(*input, **kwargs)
File "/home/tang/anaconda3/envs/ns-vqa/lib/python3.6/site-packages/torch/nn/modules/container.py", line 91, in forward
input = module(input)
File "/home/tang/anaconda3/envs/ns-vqa/lib/python3.6/site-packages/torch/nn/modules/module.py", line 491, in __call__
result = self.forward(*input, **kwargs)
File "/home/tang/anaconda3/envs/ns-vqa/lib/python3.6/site-packages/torch/nn/modules/conv.py", line 301, in forward
self.padding, self.dilation, self.groups)
RuntimeError: CUDNN_STATUS_EXECUTION_FAILED
It happened when load checkpoint
test_engine.py: 331: loading checkpoint ../../data/pretrained/object_detector.pt
Then I try the other steps or Train, the same error appears.
The system information:
Ubuntu16.04
RTX2080Ti
cuda 9.0.176
cuDNN 7.1.2
pytorch 0.4.0
python3.6.7
The code is running in the conda virtual environment.
cuda9.0 and cuDNN 7.3.1 are in base environment. are they linked with make.sh file?
I have tried many solutions from google, e.g. change version of cuda and cuDNN. But still, I got same error. I also tried other project mac-network, and it is working fine on GPU in same virtual environment.
Sincerely hope your reply!
Thanks
Hello!
I have a problem running the pretrained models.
Everything has compiled and been downloaded successfully, but when I try to run
python tools/test_net.py \ --dataset clevr_original_val \ --cfg configs/baselines/e2e_mask_rcnn_R-50-FPN_1x.yaml \ --load_ckpt ../../data/pretrained/object_detector.pt \ --output_dir ../../data/mask_rcnn/results/clevr_val_pretrained
it gives the following error:
Traceback (most recent call last):
File "tools/train_net_step.py", line 27, in
from modeling.model_builder import Generalized_RCNN
File "/home/pudu/PhD/NeSy/ns-vqa/scene_parse/mask_rcnn/lib/modeling/model_builder.py", line 11, in
from model.roi_pooling.functions.roi_pool import RoIPoolFunction
File "/home/pudu/PhD/NeSy/ns-vqa/scene_parse/mask_rcnn/lib/model/roi_pooling/functions/roi_pool.py", line 3, in
from .._ext import roi_pooling
File "/home/pudu/PhD/NeSy/ns-vqa/scene_parse/mask_rcnn/lib/model/roi_pooling/_ext/roi_pooling/init.py", line 3, in
from ._roi_pooling import lib as _lib, ffi as _ffi
ImportError: /home/pudu/PhD/NeSy/ns-vqa/scene_parse/mask_rcnn/lib/model/roi_pooling/_ext/roi_pooling/_roi_pooling.so: undefined symbol: cudaLaunchKernel
Please could somebody lend me a hand?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.