verashira / tspnet Goto Github PK

I noticed that you use WLASL I3D Model for preprocessing videos. But it in ASL not in GSL. What weight you use for this paper?

In the readme I did not find the link of your preprocessed dataset. If you intend to open source the preprocessed dataset, it would be helpful if you provide the link of the preprocessed dataset (containing the i3d features and translation texts ) in the readme file https://github.com/verashira/TSPNet#preprocessing

Requirement already satisfied: cffi in /home/shipeng/anaconda3/envs/TSPNet/lib/python3.6/site-packages (from fairseq==0.9.0) (1.14.5)
Requirement already satisfied: cython in /home/shipeng/anaconda3/envs/TSPNet/lib/python3.6/site-packages (from fairseq==0.9.0) (0.29.22)
Requirement already satisfied: numpy in /home/shipeng/anaconda3/envs/TSPNet/lib/python3.6/site-packages (from fairseq==0.9.0) (1.19.5)
Requirement already satisfied: regex in /home/shipeng/anaconda3/envs/TSPNet/lib/python3.6/site-packages (from fairseq==0.9.0) (2021.3.17)
Requirement already satisfied: sacrebleu in /home/shipeng/anaconda3/envs/TSPNet/lib/python3.6/site-packages (from fairseq==0.9.0) (1.5.1)
Requirement already satisfied: torch in /home/shipeng/anaconda3/envs/TSPNet/lib/python3.6/site-packages (from fairseq==0.9.0) (1.8.0)
Requirement already satisfied: tqdm in /home/shipeng/anaconda3/envs/TSPNet/lib/python3.6/site-packages (from fairseq==0.9.0) (4.59.0)
Requirement already satisfied: pycparser in /home/shipeng/anaconda3/envs/TSPNet/lib/python3.6/site-packages (from cffi->fairseq==0.9.0) (2.20)
Requirement already satisfied: portalocker==2.0.0 in /home/shipeng/anaconda3/envs/TSPNet/lib/python3.6/site-packages (from sacrebleu->fairseq==0.9.0) (2.0.0)
Requirement already satisfied: dataclasses in /home/shipeng/anaconda3/envs/TSPNet/lib/python3.6/site-packages (from torch->fairseq==0.9.0) (0.8)
Requirement already satisfied: typing-extensions in /home/shipeng/anaconda3/envs/TSPNet/lib/python3.6/site-packages (from torch->fairseq==0.9.0) (3.7.4.3)
Installing collected packages: fairseq
Running setup.py develop for fairseq
ERROR: Command errored out with exit status 1:
command: /home/shipeng/anaconda3/envs/TSPNet/bin/python -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/home/shipeng/project2/TSPNet/setup.py'"'"'; file='"'"'/home/shipeng/project2/TSPNet/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(file);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' develop --no-deps
cwd: /home/shipeng/project2/TSPNet/
Complete output (17 lines):
running develop
running egg_info
writing fairseq.egg-info/PKG-INFO
writing dependency_links to fairseq.egg-info/dependency_links.txt
writing entry points to fairseq.egg-info/entry_points.txt
writing requirements to fairseq.egg-info/requires.txt
writing top-level names to fairseq.egg-info/top_level.txt
reading manifest file 'fairseq.egg-info/SOURCES.txt'
writing manifest file 'fairseq.egg-info/SOURCES.txt'
running build_ext
skipping 'fairseq/data/data_utils_fast.cpp' Cython extension (up-to-date)
skipping 'fairseq/data/token_block_utils_fast.cpp' Cython extension (up-to-date)
building 'fairseq.libbleu' extension
gcc -pthread -B /home/shipeng/anaconda3/envs/TSPNet/compiler_compat -Wl,--sysroot=/ -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -stdlib=libc++ -fPIC -I/home/shipeng/anaconda3/envs/TSPNet/include/python3.6m -c fairseq/clib/libbleu/libbleu.cpp -o build/temp.linux-x86_64-3.6/fairseq/clib/libbleu/libbleu.o -std=c++11 -O3 -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE="_gcc" -DPYBIND11_STDLIB="_libstdcpp" -DPYBIND11_BUILD_ABI="_cxxabi1011" -DTORCH_EXTENSION_NAME=libbleu -D_GLIBCXX_USE_CXX11_ABI=0
/home/shipeng/anaconda3/envs/TSPNet/lib/python3.6/site-packages/torch/utils/cpp_extension.py:369: UserWarning: Attempted to use ninja as the BuildExtension backend but we could not find ninja.. Falling back to using the slow distutils backend.
warnings.warn(msg.format('we could not find ninja.'))
error: command 'gcc' failed with exit status 1
----------------------------------------
ERROR: Command errored out with exit status 1: /home/shipeng/anaconda3/envs/TSPNet/bin/python -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/home/shipeng/project2/TSPNet/setup.py'"'"'; file='"'"'/home/shipeng/project2/TSPNet/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(file);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' develop --no-deps Check the logs for full command output.

I have installed gcc,

I am trying to extract I3D feature for my dataset, but not understanding how to do it. Can you guide me about this?

RGB or flow, which feature of I3D did you use? Can you please confirm?

When creating the target dataset (indexed dataset), we need to load the index_file_path (eg. "data-bin/test.sign-en.en"+".idx") and data_file_path (eg."data-bin/test.sign-en.en"+".bin").
However, these two files are not mentioned in README, may I know how to generate them, and what's their purpose?

Hello, Xu

I am interested in using your model for my project, but I noticed that the model weight is not available in the repository. Could you please provide me with the link to download the model weight or upload it to the repository?

Thank you for your time and effort.

I want to use my own video dataset.

I make a text preprocessing but how can i make a video feature by using I3D networks?

Traceback (most recent call last):
File "test_scripts/test_sign_local.py", line 246, in
cli_main()
File "test_scripts/test_sign_local.py", line 235, in cli_main
torch.multiprocessing.spawn(
File "/home/lee/anaconda3/envs/torch/lib/python3.8/site-packages/torch/multiprocessing/spawn.py", line 199, in spawn
return start_processes(fn, args, nprocs, join, daemon, start_method='spawn')
File "/home/lee/anaconda3/envs/torch/lib/python3.8/site-packages/torch/multiprocessing/spawn.py", line 157, in start_processes
while not context.join():
File "/home/lee/anaconda3/envs/torch/lib/python3.8/site-packages/torch/multiprocessing/spawn.py", line 118, in join
raise Exception(msg)
Exception:

-- Process 3 terminated with the following error:
Traceback (most recent call last):
File "/home/lee/anaconda3/envs/torch/lib/python3.8/site-packages/torch/multiprocessing/spawn.py", line 19, in _wrap
fn(i, *args)
File "/home/lee/Signlanguage/TSPNet-main/test_scripts/test_sign_local.py", line 176, in distributed_main
main(args, init_distributed=True)
File "/home/lee/Signlanguage/TSPNet-main/test_scripts/test_sign_local.py", line 45, in main
args.distributed_rank = distributed_utils.distributed_init(args)
File "/home/lee/Signlanguage/TSPNet-main/fairseq/distributed_utils.py", line 98, in distributed_init
dist.all_reduce(torch.zeros(1).cuda())
File "/home/lee/anaconda3/envs/torch/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py", line 941, in all_reduce
work = _default_pg.allreduce([tensor], opts)
RuntimeError: Socket Timeout

I run the code but it has a problem... how can i do it? i need your guide or help thank you

Can you provide the link of preprocessed data for downloading?

epoch 001:   0% 0/111 [00:00<?, ?it/s]/content/drive/My Drive/TSPNet/fairseq/optim/adam.py:179: UserWarning: This overload of add_ is deprecated:
	add_(Number alpha, Tensor other)
Consider using one of the following signatures instead:
	add_(Tensor other, *, Number alpha) (Triggered internally at  /pytorch/torch/csrc/utils/python_arg_parser.cpp:882.)
  exp_avg.mul_(beta1).add_(1 - beta1, grad)
epoch 001 | loss 9.643 | nll_loss 9.33 | ppl 643.732 | wps 22.2 | ups 0.02 | wpb 1212.8 | bsz 63.9 | num_updates 111 | lr 0.0001 | gnorm 1.309 | clip 0 | oom 0 | train_wall 5912 | wall 6050
epoch 001 | valid on 'test' subset:   0% 0/10 [00:00<?, ?it/s]Traceback (most recent call last):
  File "train.py", line 11, in <module>
    cli_main()
  File "/content/drive/My Drive/TSPNet/fairseq_cli/train.py", line 323, in cli_main
    main(args)
  File "/content/drive/My Drive/TSPNet/fairseq_cli/train.py", line 105, in main
    valid_losses = validate(args, trainer, task, epoch_itr, valid_subsets)
  File "/content/drive/My Drive/TSPNet/fairseq_cli/train.py", line 250, in validate
    trainer.valid_step(sample)
  File "/usr/lib/python3.6/contextlib.py", line 52, in inner
    return func(*args, **kwds)
  File "/content/drive/My Drive/TSPNet/fairseq/trainer.py", line 452, in valid_step
    raise e
  File "/content/drive/My Drive/TSPNet/fairseq/trainer.py", line 437, in valid_step
    sample, self.model, self.criterion
  File "/content/drive/My Drive/TSPNet/fairseq/tasks/translation_sign.py", line 239, in valid_step
    bleu, hyps, refs = self._inference_with_bleu(self.sequence_generator, sample, model)
  File "/content/drive/My Drive/TSPNet/fairseq/tasks/translation_sign.py", line 304, in _inference_with_bleu
    gen_out = self.inference_step(generator, [model], sample, None)
  File "/content/drive/My Drive/TSPNet/fairseq/tasks/fairseq_task.py", line 309, in inference_step
    return generator.generate(models, sample, prefix_tokens=prefix_tokens)
  File "/usr/local/lib/python3.6/dist-packages/torch/autograd/grad_mode.py", line 26, in decorate_context
    return func(*args, **kwargs)
  File "/content/drive/My Drive/TSPNet/fairseq/sequence_generator.py", line 92, in generate
    return self._generate(model, sample, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/torch/autograd/grad_mode.py", line 26, in decorate_context
    return func(*args, **kwargs)
  File "/content/drive/My Drive/TSPNet/fairseq/sequence_generator.py", line 361, in _generate
    scores.view(bsz, beam_size, -1)[:, :, :step],
  File "/content/drive/My Drive/TSPNet/fairseq/search.py", line 81, in step
    torch.div(self.indices_buf, vocab_size, out=self.beams_buf)

RuntimeError: result type Float can't be cast to the desired output type Long

I was trying to run the code in Google Colaboratory but I am getting this error. Can you guide me on how to solve this?

data-bin/
├── train.sign-de.sign
├── train.sign-de.de
│
├── test.sign-de.sign
├── test.sign-de.de
│
├── emb
└── dict.de.txt

in your data-bin, what is the emb file and how can I generate for my data?

I want to directly test a gesture video, and then output text, how to implement the code