verashira / tspnet Goto Github PK
View Code? Open in Web Editor NEWTSPNet: Hierarchical Feature Learning via Temporal Semantic Pyramid for Sign Language Translation
License: MIT License
TSPNet: Hierarchical Feature Learning via Temporal Semantic Pyramid for Sign Language Translation
License: MIT License
I noticed that you use WLASL I3D Model for preprocessing videos. But it in ASL not in GSL. What weight you use for this paper?
In the readme I did not find the link of your preprocessed dataset. If you intend to open source the preprocessed dataset, it would be helpful if you provide the link of the preprocessed dataset (containing the i3d features and translation texts ) in the readme file https://github.com/verashira/TSPNet#preprocessing
Requirement already satisfied: cffi in /home/shipeng/anaconda3/envs/TSPNet/lib/python3.6/site-packages (from fairseq==0.9.0) (1.14.5)
Requirement already satisfied: cython in /home/shipeng/anaconda3/envs/TSPNet/lib/python3.6/site-packages (from fairseq==0.9.0) (0.29.22)
Requirement already satisfied: numpy in /home/shipeng/anaconda3/envs/TSPNet/lib/python3.6/site-packages (from fairseq==0.9.0) (1.19.5)
Requirement already satisfied: regex in /home/shipeng/anaconda3/envs/TSPNet/lib/python3.6/site-packages (from fairseq==0.9.0) (2021.3.17)
Requirement already satisfied: sacrebleu in /home/shipeng/anaconda3/envs/TSPNet/lib/python3.6/site-packages (from fairseq==0.9.0) (1.5.1)
Requirement already satisfied: torch in /home/shipeng/anaconda3/envs/TSPNet/lib/python3.6/site-packages (from fairseq==0.9.0) (1.8.0)
Requirement already satisfied: tqdm in /home/shipeng/anaconda3/envs/TSPNet/lib/python3.6/site-packages (from fairseq==0.9.0) (4.59.0)
Requirement already satisfied: pycparser in /home/shipeng/anaconda3/envs/TSPNet/lib/python3.6/site-packages (from cffi->fairseq==0.9.0) (2.20)
Requirement already satisfied: portalocker==2.0.0 in /home/shipeng/anaconda3/envs/TSPNet/lib/python3.6/site-packages (from sacrebleu->fairseq==0.9.0) (2.0.0)
Requirement already satisfied: dataclasses in /home/shipeng/anaconda3/envs/TSPNet/lib/python3.6/site-packages (from torch->fairseq==0.9.0) (0.8)
Requirement already satisfied: typing-extensions in /home/shipeng/anaconda3/envs/TSPNet/lib/python3.6/site-packages (from torch->fairseq==0.9.0) (3.7.4.3)
Installing collected packages: fairseq
Running setup.py develop for fairseq
ERROR: Command errored out with exit status 1:
command: /home/shipeng/anaconda3/envs/TSPNet/bin/python -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/home/shipeng/project2/TSPNet/setup.py'"'"'; file='"'"'/home/shipeng/project2/TSPNet/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(file);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' develop --no-deps
cwd: /home/shipeng/project2/TSPNet/
Complete output (17 lines):
running develop
running egg_info
writing fairseq.egg-info/PKG-INFO
writing dependency_links to fairseq.egg-info/dependency_links.txt
writing entry points to fairseq.egg-info/entry_points.txt
writing requirements to fairseq.egg-info/requires.txt
writing top-level names to fairseq.egg-info/top_level.txt
reading manifest file 'fairseq.egg-info/SOURCES.txt'
writing manifest file 'fairseq.egg-info/SOURCES.txt'
running build_ext
skipping 'fairseq/data/data_utils_fast.cpp' Cython extension (up-to-date)
skipping 'fairseq/data/token_block_utils_fast.cpp' Cython extension (up-to-date)
building 'fairseq.libbleu' extension
gcc -pthread -B /home/shipeng/anaconda3/envs/TSPNet/compiler_compat -Wl,--sysroot=/ -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -stdlib=libc++ -fPIC -I/home/shipeng/anaconda3/envs/TSPNet/include/python3.6m -c fairseq/clib/libbleu/libbleu.cpp -o build/temp.linux-x86_64-3.6/fairseq/clib/libbleu/libbleu.o -std=c++11 -O3 -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE="_gcc" -DPYBIND11_STDLIB="_libstdcpp" -DPYBIND11_BUILD_ABI="_cxxabi1011" -DTORCH_EXTENSION_NAME=libbleu -D_GLIBCXX_USE_CXX11_ABI=0
/home/shipeng/anaconda3/envs/TSPNet/lib/python3.6/site-packages/torch/utils/cpp_extension.py:369: UserWarning: Attempted to use ninja as the BuildExtension backend but we could not find ninja.. Falling back to using the slow distutils backend.
warnings.warn(msg.format('we could not find ninja.'))
error: command 'gcc' failed with exit status 1
----------------------------------------
ERROR: Command errored out with exit status 1: /home/shipeng/anaconda3/envs/TSPNet/bin/python -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/home/shipeng/project2/TSPNet/setup.py'"'"'; file='"'"'/home/shipeng/project2/TSPNet/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(file);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' develop --no-deps Check the logs for full command output.
I have installed gcc,
I am trying to extract I3D feature for my dataset, but not understanding how to do it. Can you guide me about this?
RGB or flow, which feature of I3D did you use? Can you please confirm?
When creating the target dataset (indexed dataset), we need to load the index_file_path (eg. "data-bin/test.sign-en.en"+".idx") and data_file_path (eg."data-bin/test.sign-en.en"+".bin").
However, these two files are not mentioned in README, may I know how to generate them, and what's their purpose?
Hello, Xu
I am interested in using your model for my project, but I noticed that the model weight is not available in the repository. Could you please provide me with the link to download the model weight or upload it to the repository?
Thank you for your time and effort.
I want to use my own video dataset.
I make a text preprocessing but how can i make a video feature by using I3D networks?
Traceback (most recent call last):
File "test_scripts/test_sign_local.py", line 246, in
cli_main()
File "test_scripts/test_sign_local.py", line 235, in cli_main
torch.multiprocessing.spawn(
File "/home/lee/anaconda3/envs/torch/lib/python3.8/site-packages/torch/multiprocessing/spawn.py", line 199, in spawn
return start_processes(fn, args, nprocs, join, daemon, start_method='spawn')
File "/home/lee/anaconda3/envs/torch/lib/python3.8/site-packages/torch/multiprocessing/spawn.py", line 157, in start_processes
while not context.join():
File "/home/lee/anaconda3/envs/torch/lib/python3.8/site-packages/torch/multiprocessing/spawn.py", line 118, in join
raise Exception(msg)
Exception:
-- Process 3 terminated with the following error:
Traceback (most recent call last):
File "/home/lee/anaconda3/envs/torch/lib/python3.8/site-packages/torch/multiprocessing/spawn.py", line 19, in _wrap
fn(i, *args)
File "/home/lee/Signlanguage/TSPNet-main/test_scripts/test_sign_local.py", line 176, in distributed_main
main(args, init_distributed=True)
File "/home/lee/Signlanguage/TSPNet-main/test_scripts/test_sign_local.py", line 45, in main
args.distributed_rank = distributed_utils.distributed_init(args)
File "/home/lee/Signlanguage/TSPNet-main/fairseq/distributed_utils.py", line 98, in distributed_init
dist.all_reduce(torch.zeros(1).cuda())
File "/home/lee/anaconda3/envs/torch/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py", line 941, in all_reduce
work = _default_pg.allreduce([tensor], opts)
RuntimeError: Socket Timeout
I run the code but it has a problem... how can i do it? i need your guide or help thank you
Can you provide the link of preprocessed data for downloading?
epoch 001: 0% 0/111 [00:00<?, ?it/s]/content/drive/My Drive/TSPNet/fairseq/optim/adam.py:179: UserWarning: This overload of add_ is deprecated:
add_(Number alpha, Tensor other)
Consider using one of the following signatures instead:
add_(Tensor other, *, Number alpha) (Triggered internally at /pytorch/torch/csrc/utils/python_arg_parser.cpp:882.)
exp_avg.mul_(beta1).add_(1 - beta1, grad)
epoch 001 | loss 9.643 | nll_loss 9.33 | ppl 643.732 | wps 22.2 | ups 0.02 | wpb 1212.8 | bsz 63.9 | num_updates 111 | lr 0.0001 | gnorm 1.309 | clip 0 | oom 0 | train_wall 5912 | wall 6050
epoch 001 | valid on 'test' subset: 0% 0/10 [00:00<?, ?it/s]Traceback (most recent call last):
File "train.py", line 11, in <module>
cli_main()
File "/content/drive/My Drive/TSPNet/fairseq_cli/train.py", line 323, in cli_main
main(args)
File "/content/drive/My Drive/TSPNet/fairseq_cli/train.py", line 105, in main
valid_losses = validate(args, trainer, task, epoch_itr, valid_subsets)
File "/content/drive/My Drive/TSPNet/fairseq_cli/train.py", line 250, in validate
trainer.valid_step(sample)
File "/usr/lib/python3.6/contextlib.py", line 52, in inner
return func(*args, **kwds)
File "/content/drive/My Drive/TSPNet/fairseq/trainer.py", line 452, in valid_step
raise e
File "/content/drive/My Drive/TSPNet/fairseq/trainer.py", line 437, in valid_step
sample, self.model, self.criterion
File "/content/drive/My Drive/TSPNet/fairseq/tasks/translation_sign.py", line 239, in valid_step
bleu, hyps, refs = self._inference_with_bleu(self.sequence_generator, sample, model)
File "/content/drive/My Drive/TSPNet/fairseq/tasks/translation_sign.py", line 304, in _inference_with_bleu
gen_out = self.inference_step(generator, [model], sample, None)
File "/content/drive/My Drive/TSPNet/fairseq/tasks/fairseq_task.py", line 309, in inference_step
return generator.generate(models, sample, prefix_tokens=prefix_tokens)
File "/usr/local/lib/python3.6/dist-packages/torch/autograd/grad_mode.py", line 26, in decorate_context
return func(*args, **kwargs)
File "/content/drive/My Drive/TSPNet/fairseq/sequence_generator.py", line 92, in generate
return self._generate(model, sample, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/torch/autograd/grad_mode.py", line 26, in decorate_context
return func(*args, **kwargs)
File "/content/drive/My Drive/TSPNet/fairseq/sequence_generator.py", line 361, in _generate
scores.view(bsz, beam_size, -1)[:, :, :step],
File "/content/drive/My Drive/TSPNet/fairseq/search.py", line 81, in step
torch.div(self.indices_buf, vocab_size, out=self.beams_buf)
RuntimeError: result type Float can't be cast to the desired output type Long
I was trying to run the code in Google Colaboratory but I am getting this error. Can you guide me on how to solve this?
data-bin/
├── train.sign-de.sign
├── train.sign-de.de
│
├── test.sign-de.sign
├── test.sign-de.de
│
├── emb
└── dict.de.txt
in your data-bin, what is the emb file and how can I generate for my data?
I want to directly test a gesture video, and then output text, how to implement the code
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.