sairin1202 / commonsense-knowledge-aware-concept-selection-for-diverse-and-informative-visual-storytelling Goto Github PK

View Code? Open in Web Editor NEW

12.0 3.0 2.0 72.69 MB

The implement of Commonsense Knowledge Aware Concept Selection For Diverse and Informative Visual Storytelling

License: MIT License

Python 100.00%

commonsense-knowledge-aware-concept-selection-for-diverse-and-informative-visual-storytelling's Introduction

Commonsense-Knowledge-Aware-Concept-Selection-For-Diverse-and-Informative-Visual-Storytelling

The implement of Commonsense Knowledge Aware Concept Selection For Diverse and Informative Visual Storytelling

This repository consists of two parts:

Concept Selection
Visual Storytelling

Concept Selection

In the paper, we use the concept selection to first select the concepts for the story generation.

codes are involved under concept_selection/

run python train.py to train the model.

Visual Storytelling

In the paper, we use the concepts and the image as inputs to generate the story.

We use BART as pretrained model.

codes are involved under visualstorytelling/

run python train.py to train the model.

Data

All the data used in the project can be found in https://drive.google.com/file/d/1jW0fgZnCUX3EIefwTcR0l8bHDrgDSXwm/view?usp=sharing

commonsense-knowledge-aware-concept-selection-for-diverse-and-informative-visual-storytelling's People

Contributors

Stargazers

Watchers

Forkers

strategist922 lzc2017

commonsense-knowledge-aware-concept-selection-for-diverse-and-informative-visual-storytelling's Issues

Some essential files are missing!

Thanks for your excellent work! But some essential files are missing(et al. data/train.pkl, data/test.pkl, data/clarifai/), in addition, could you complete the README.md to give more detailed instructions about data processing, training and testing. Appreciate it if you could reply.

The paper description differs from the provided code implementation？

Hello, I noticed in your paper when I reproduced your results that you say "During training, we follow Jung et al. (2020) to blind one of the images starting from the 50-th epoch and increase the blinding into two images from epoch 80. The training stops at epoch 100.", but I don't see this part of the code in the code you provided, is it possible to add it if possible? In your paper, you said "VIST dataset (Huang et al. 2016), which consists of 40101 samples for training 5050 test samples. and one corresponding gold story." but the results you provided only have 1010, why is that?

About test_reference.json

Hello, in test_reference.json one album_id corresponds to 5 stories, but in the test process you only generate one story for each album_id, why do you use test_reference.json in evaluation.py to evaluate the generated results?

VIST data

Hi! The VIST dataset provides five stories and three captions for each image, but I just need one of each them. Could you tell how to select the stroy and the caption from the dataset? Is it random selection?

sorry to bother you,but I found this link has gone stale https://drive.google.com/file/d/1jW0fgZnCUX3EIefwTcR0l8bHDrgDSXwm/view?usp=sharing,could you provide us the data,thanks very much!

When I run the train.py in visualstorytelling, errors happens as list, do you have the solution?:

Downloading: "https://github.com/pytorch/fairseq/archive/master.zip" to /root/.cache/torch/hub/master.zip
Traceback (most recent call last):
File "/opt/data/private/test14/liushidi/code/a.py", line 2, in
bart = torch.hub.load('pytorch/fairseq', 'bart.large')
File "/opt/conda/envs/liushidi/lib/python3.7/site-packages/torch/hub.py", line 368, in load
repo_or_dir = _get_cache_or_reload(repo_or_dir, force_reload, verbose)
File "/opt/conda/envs/liushidi/lib/python3.7/site-packages/torch/hub.py", line 141, in _get_cache_or_reload
download_url_to_file(url, cached_file, progress=False)
File "/opt/conda/envs/liushidi/lib/python3.7/site-packages/torch/hub.py", line 425, in download_url_to_file
u = urlopen(req)
File "/opt/conda/envs/liushidi/lib/python3.7/urllib/request.py", line 222, in urlopen
return opener.open(url, data, timeout)
File "/opt/conda/envs/liushidi/lib/python3.7/urllib/request.py", line 525, in open
response = self._open(req, data)
File "/opt/conda/envs/liushidi/lib/python3.7/urllib/request.py", line 543, in _open
'_open', req)
File "/opt/conda/envs/liushidi/lib/python3.7/urllib/request.py", line 503, in _call_chain
result = func(*args)
File "/opt/conda/envs/liushidi/lib/python3.7/urllib/request.py", line 1393, in https_open
context=self._context, check_hostname=self._check_hostname)
File "/opt/conda/envs/liushidi/lib/python3.7/urllib/request.py", line 1353, in do_open
r = h.getresponse()
File "/opt/conda/envs/liushidi/lib/python3.7/http/client.py", line 1373, in getresponse
response.begin()
File "/opt/conda/envs/liushidi/lib/python3.7/http/client.py", line 319, in begin
version, status, reason = self._read_status()
File "/opt/conda/envs/liushidi/lib/python3.7/http/client.py", line 280, in _read_status
line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
File "/opt/conda/envs/liushidi/lib/python3.7/socket.py", line 589, in readinto
return self._sock.recv_into(b)
File "/opt/conda/envs/liushidi/lib/python3.7/ssl.py", line 1071, in recv_into
return self.read(nbytes, buffer)
File "/opt/conda/envs/liushidi/lib/python3.7/ssl.py", line 929, in read
return self._sslobj.read(len, buffer)
ssl.SSLError: [SSL: KRB5_S_TKT_NYV] unexpected eof while reading (_ssl.c:2570)

When to clean the code?

RuntimeError: cannot train after make_generation_fast

When I run the train.py in visualstorytelling, errors happens as list, do you have the solution?:

Traceback (most recent call last):
  File "train.py", line 109, in <module>
    main()
  File "train.py", line 105, in main
    train_model(opt, bart, dataset, test_dataset, 10)
  File "train.py", line 42, in train_model
    bart._model.train()
  File "/home/caidesheng/miniconda3/envs/teran/lib/python3.6/site-packages/torch/nn/modules/module.py", line 1277, in train
    module.train(mode)
  File "/home/caidesheng/miniconda3/envs/teran/lib/python3.6/site-packages/torch/nn/modules/module.py", line 1277, in train
    module.train(mode)
  File "/home/caidesheng/miniconda3/envs/teran/lib/python3.6/site-packages/torch/nn/modules/module.py", line 1277, in train
    module.train(mode)
  File "/home/caidesheng/.cache/torch/hub/pytorch_fairseq_master/fairseq/models/fairseq_model.py", line 212, in train
    raise RuntimeError("cannot train after make_generation_fast")
RuntimeError: cannot train after make_generation_fast

it seems due to the code in fairseq_model.py of fairseq:

apply_make_generation_fast_(self, "")
def train(mode=True):
   if mode:
        raise RuntimeError("cannot train after make_generation_fast")

# this model should no longer be used for training
self.eval()
self.train = train

TypeError: tuple indices must be integers or slices, not str

Hello, Now I have an error when I run the evaluation.py in visualstorytelling section, it seems due to the definition of EncoderOut in visualstorytelling/generator.py which is a namedtuple. error is as follows:

Traceback (most recent call last):
  File "evaluation.py", line 169, in <module>
    main()
  File "evaluation.py", line 165, in main
    generate(opt, bart)
  File "evaluation.py", line 135, in generate
    hypo = generate_sentence(album_id, keywords, model)  # keywords  5 ge 809
  File "evaluation.py", line 90, in generate_sentence
    story = model.generate(cond=feats, keys=keywords, top_k=-1, top_p=0.9)
  File "/home/caidesheng/caojie/workspace2/Commonsense-Knowledge-Aware-Concept-Selection-For-Diverse-and-Informative-Visual-Storytelling/visualstorytelling/bart.py", line 109, in generate
    'src_lengths': torch.tensor([cond.shape[1]+key_tokens.shape[1]])
  File "/home/caidesheng/miniconda3/envs/teran/lib/python3.6/site-packages/torch/autograd/grad_mode.py", line 26, in decorate_context
    return func(*args, **kwargs)
  File "/home/caidesheng/caojie/workspace2/Commonsense-Knowledge-Aware-Concept-Selection-For-Diverse-and-Informative-Visual-Storytelling/visualstorytelling/generator.py", line 116, in generate
    return self._generate(model, sample, **kwargs)
  File "/home/caidesheng/miniconda3/envs/teran/lib/python3.6/site-packages/torch/autograd/grad_mode.py", line 26, in decorate_context
    return func(*args, **kwargs)
  File "/home/caidesheng/caojie/workspace2/Commonsense-Knowledge-Aware-Concept-Selection-For-Diverse-and-Informative-Visual-Storytelling/visualstorytelling/generator.py", line 304, in _generate
    tokens[:, :step + 1], encoder_outs, temperature=self.temperature,
  File "/home/caidesheng/miniconda3/envs/teran/lib/python3.6/site-packages/torch/autograd/grad_mode.py", line 26, in decorate_context
    return func(*args, **kwargs)
  File "/home/caidesheng/caojie/workspace2/Commonsense-Knowledge-Aware-Concept-Selection-For-Diverse-and-Informative-Visual-Storytelling/visualstorytelling/generator.py", line 545, in forward_decoder
    temperature=temperature,
  File "/home/caidesheng/caojie/workspace2/Commonsense-Knowledge-Aware-Concept-Selection-For-Diverse-and-Informative-Visual-Storytelling/visualstorytelling/generator.py", line 577, in _decode_one
    if self.incremental_states is not None:
  File "/home/caidesheng/.cache/torch/hub/pytorch_fairseq_master/fairseq/models/fairseq_model.py", line 326, in forward_decoder
    return self.decoder(prev_output_tokens, **kwargs)
  File "/home/caidesheng/miniconda3/envs/teran/lib/python3.6/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/caidesheng/.cache/torch/hub/pytorch_fairseq_master/fairseq/models/transformer/transformer_decoder.py", line 222, in forward
    alignment_heads=alignment_heads,
  File "/home/caidesheng/.cache/torch/hub/pytorch_fairseq_master/fairseq/models/transformer/transformer_decoder.py", line 244, in extract_features
    alignment_heads,
  File "/home/caidesheng/.cache/torch/hub/pytorch_fairseq_master/fairseq/models/transformer/transformer_decoder.py", line 287, in extract_features_scriptable
    if encoder_out is not None and len(encoder_out["encoder_out"]) > 0:
TypeError: tuple indices must be integers or slices, not str

How to reproduce automatic metric in the paper.

Hi,
Thanks for sharing the amazing work, we observe that you do not seem to give the automatic evaluation score of BART model in the paper. And we calculated the score of stories in result/result.csv, which is lower than MCSM+RNN score in the paper.
Could you share us the script to generate stories using MCSM+RNN model, or tell us what automatic evaluation score the BART model should have.
Thanks a lot.