yl3800 / transtr Goto Github PK

Thank you for sharing your projects. When I train TranSTR on the nextqa dataset, it achieved 60.35 on the test set. How can I further improve the accuracy to 61.5 ?
Here is my training log.

First of all, I would like to thank you for open-sourcing your code for TranSTR. It is a valuable resource for the research community.

I am trying to use your code, but I am facing an issue with the pre-processing of the video features.

The code does not provide any pre-processing code, so I am using my own video features, which are 2048 dimensions. When I try to use these features in the multimodal_transformer.py file, I am getting an error in the with_pos_embed function. The error is because the pos variable is 768 dimensions, but my video features are 2048 dimensions.

I would like to request that you provide some pre-processing code for the video features. This would make it easier for users to use your code with their own video features.

I would appreciate any help you can provide with this issue.

Hello, thank you for sharing the code. Please tell me how I should process the characteristics of the data. I don't understand what the path refers to here:
sample_list_path="/data/vqa/causal/anno"
video_feature_path="/region_feat_aln"
frame_feat_file = osp.join('/model_base_vqa_capfilt_large', '{}.h5'.format(split))

object_feat_file = osp.join(video_feature_path, "region_feat_n/acregion_8c20b_{}.h5".format(split))
frame_feat_file = osp.join('/storage_fast/ycli/data/vqa/next/feature/blip/model_base_vqa_capfilt_large',
'{}.h5'.format(split))
sample_list_path="/storage_fast/ycli/data/vqa/next/anno"

Hello，thanks for sharing the code，I try to run the code but the files of dataset missing，especially MSVD and MSRVTT.
I try to read the datasetloader but find that the filepath and filename of files in need is strage，such as “acregion_8c10b_{}.h5” which contains object feather and “/msvd/anno/ans_word.json” annotation file. It is hard for me to guess how to get these files and the format of data in them. So could you please share the files or info us about the method to get these files？ I‘ll be grateful if you can notice this message and answer us.

When I run the train.py with the default set, I met the error "RuntimeError: CUDA error: an illegal memory access was encountered". How can I solve it?

Hello, thank you for sharing the code. Please tell me what the path refers to the Causal-VidQA （which I got ['data/visual_feature'], ['data/text_feature'], ['data/split'] and ['data/QA'] in https://github.com/bcmi/Causal-VidQA/tree/master）
class VideoQADataset(Dataset):
def init(self, split, n_query=5, obj_num=1, sample_list_path="/data/vqa/causal/anno",
video_feature_path="/region_feat_aln" ):
super(VideoQADataset, self).init()
# 读取dataset
self.sample_list_file = osp.join(sample_list_path, "{}.csv".format(split))
self.sample_list = load_file(self.sample_list_file)
self.split = split
self.mc = n_query
self.obj_num = obj_num
# 读取video feature

    frame_feat_file = osp.join('/model_base_vqa_capfilt_large', '{}.h5'.format(split))
    # object_feat_file = osp.join(video_feature_path, "region_feat_n/acregion_8c20b_{}.h5".format(split))
    self.map_dir = load_file(osp.join(sample_list_path, 'map_dir_caul.json'))
    self.obj_feat_dir = video_feature_path

Thank you for sharing your code! I want to reproduce on the nextqa dataset, but I can not find the model_base_vqa_capfilt_large h5 files which are required. Can you share these? Additionally, can you share your environment requirements with me?

The requirement file contains some invalid command such as “backcall @ file:///home/conda/feedstock_root/build_artifacts/backcall_1592338393461/work”。Could you share a refined requirement file? Thank you.