mathux / actor Goto Github PK

Hi Mathis
Thanks for sharing the implementation. I have two issues:

1. If I understand correctly, this work can be treated as a sequence VAE model used to learn distributions from observations. The KL-divergence loss term encourages the learned distribution to be as close as to a normal distribution. So have you ever encountered the KL-vanishing (KL loss = 0) issue? If yes, how have you solved it ?

2. Instead of feeding the sampled code z merely to Transformer decoder block, how about to adding it to all tokens for the decoder input sequence (plus with the position embeddings)?

Thanks

Hi, thank you for the code, I have already trained some models and would like to visualize the results. However, the scripts visualize_latent_space.py and visualize_sequence.py are trying to import the module optutils, which seems to be missing from the repository. Could you please add this module?

Hello, it's a great work. But I have a question about this paper. This paper gave me a lot of inspiration. For the Decoder, directly generating all the results at one time will make the training process difficult. So what method is used in the paper to ensure that the decoder is provided with a inductive bias to ensure that it can perform well during training?

Thanks for the wonderful work @Mathux!
I really enjoyed reading paper and also testing it on my environment,
while looking at your code and paper, I found out that you've set dimensionality of embedding for VAE to 256 and I'm wondering if there's any reason for this.

Thanks,
Joseph

Wonderful work! Now, I have a question about how to convert 3D keypoints estimated by other models and transfer them to the desired mesh by parameters of mean meshed SMPL. Please forgive I'm a beginer for learning SMPL.

Hi, thanks for your great work!
I'm interested in concatenating two actions using your model. But I'm not sure what's means of "interpolation in latent space".
Could you give more details or some guidance?

Hi,
Thanks for releasing your code. I am wondering what is the train/val/test split that you are using for HumanAct12? It seems that you are always using the entire dataset for training and evaluating if I am not mistaken..
Tanks for your feedback and your help.

Hello Dear Author. Thank you very much for your good work. I wanted to ask if there is a way to add clothes, skin, background to these 3d actions.

Thank you very much,

Sardor

Hi, @Mathux , Thanks for your nice work!
I encountered some issue when i generated some motion sequences.
Like the shake problem occurred about 60 frames later, do you have any idea about it? thanks!

Hi

Thanks for the repo and the documentation!

How can I create .pkl files for a custom dataset to train models on or evaluate current models on them?

Hello,

I am an University student at UHA (Université de Haute Alsace) and as my university project,
I have to try to make your code ACTOR to work on Google Colab.
But when I got to the part when i tried to visualize the grid I got this error:

I got the same error with the trained models created with the command train or the pretrained models.

Could you please tell me if you have a solution to this problem?

Cordialy,
Scherrer Xavier

I want to expand output length like following method,
i.e. run decoder twice.
Because i think if i only expand length from 60 to 120 lengths, meaningless output will be generated after 60th frame.

If i generate 120 length(60 length *2)
input -> decoder -> output(60 length) -> input -> decoder -> output(60 length)
=> total 120 length.

when i run python -m src.render.rendermotion exps/humanact12/generation.npy

i got 'FileNotFoundError: [Errno 2] No such file or directory: 'models/smpl/smplfaces.npy''

Hello,Mathux! Thanks for your implementation. When I tried to render motions with generation.npy， the error occurs like the following. I have tried the introduction in this link mmatl/pyrender#86, but it does not work. Thank you in advance!

Hi,

Is there a way to store model generated rotations as bvh file ?
I want to load the generated motion via following package in order to proceed with my research.
Thanks.

Could anyone tell me what the 13 action type in NTURGB120 is used?
I appreciate any reply!
THX

Hi @Mathux !
Can you share with us the training commands reproducing the gru and fc models reported in your paper? Thanks:)

xseq = torch.cat((self.muQuery[y][None], self.sigmaQuery[y][None], x), axis=0)

This code tries to concatenate 2d and 3d tensors.
Can you explain how it works?

Thanks in advance.

hello, when I want to train stgcn using new dataset, I find that there is no ‘src.util.trainer’， did the author forget to put it in？

Hi,
first of all, Thank you for releasing the nicely arranged code.
Released code really helped understanding of the model, and I appreciate kindly written instructions for the code.

However, I encountered undesirable result on UETSC dataset, which comes from the model I trained from the scratch.
I followed instruction on the dataset, also did not modify training script.
python -m src.train.train_cvae --modelname cvae_transformer_rc_rcxyz_kl --pose_rep rot6d --lambda_kl 1e-5 --jointstype vertices --batch_size 20 --num_frames 60 --num_layers 8 --lr 0.0001 --glob --translation --no-vertstrans --dataset uestc --num_epochs 1000 --snapshot 100 --folder exps/uestc

I double-checked whether saved opt.yml file from the experiment is identical to the pretrained model, but I found that no configurations were different.
Also, I conducted the same experiment twice but results were identical.

This is the train log of the released pretrained model,

and these are the train logs from the trainings I conducted.

As my training result on HumanAct dataset is same with the pre-trained model, I guess no training environmental issue involves here.

Is there any possible reason for this failure of the training?

Hello Author thank you for the good work. I can say it is impressive. But while running your code I came across to an error.

Thank you beforehand for the reply!

When I try to render the generated motions. I got the following error:

(actor) nankaingy@lza-PC:~/data/code/pose-shape/ACTOR$ python -m src.render.rendermotion pretrained_models/humanact12/generation.npy
Visualize generation_0, action 0: 0%| | 0/60 [00:00<?, ?it/s]
Traceback (most recent call last):
File "/home/nankaingy/data/anaconda3/envs/actor/lib/python3.8/site-packages/OpenGL/latebind.py", line 41, in call
return self._finalCall( *args, **named )
TypeError: 'NoneType' object is not callable

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/nankaingy/data/anaconda3/envs/actor/lib/python3.8/runpy.py", line 194, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/home/nankaingy/data/anaconda3/envs/actor/lib/python3.8/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "/home/nankaingy/data/code/pose-shape/ACTOR/src/render/rendermotion.py", line 92, in
main()
File "/home/nankaingy/data/code/pose-shape/ACTOR/src/render/rendermotion.py", line 88, in main
render_video(meshes, key, action, renderer, path, background)
File "/home/nankaingy/data/code/pose-shape/ACTOR/src/render/rendermotion.py", line 27, in render_video
img = renderer.render(background, mesh, cam, color=color)
File "/home/nankaingy/data/code/pose-shape/ACTOR/src/render/renderer.py", line 143, in render
rgb, _ = self.renderer.render(self.scene, flags=render_flags)
File "/home/nankaingy/data/anaconda3/envs/actor/lib/python3.8/site-packages/pyrender/offscreen.py", line 99, in render
return self._renderer.render(scene, flags)
File "/home/nankaingy/data/anaconda3/envs/actor/lib/python3.8/site-packages/pyrender/renderer.py", line 121, in render
self._update_context(scene, flags)
File "/home/nankaingy/data/anaconda3/envs/actor/lib/python3.8/site-packages/pyrender/renderer.py", line 709, in _update_context
p._add_to_context()
File "/home/nankaingy/data/anaconda3/envs/actor/lib/python3.8/site-packages/pyrender/primitive.py", line 324, in _add_to_context
self._vaid = glGenVertexArrays(1)
File "/home/nankaingy/data/anaconda3/envs/actor/lib/python3.8/site-packages/OpenGL/latebind.py", line 45, in call
return self._finalCall( *args, **named )
File "/home/nankaingy/data/anaconda3/envs/actor/lib/python3.8/site-packages/OpenGL/wrapper.py", line 657, in wrapperCall
result = wrappedOperation( *cArguments )
File "/home/nankaingy/data/anaconda3/envs/actor/lib/python3.8/site-packages/OpenGL/platform/baseplatform.py", line 401, in call
if self.load():
File "/home/nankaingy/data/anaconda3/envs/actor/lib/python3.8/site-packages/OpenGL/platform/baseplatform.py", line 383, in load
func = platform.PLATFORM.constructFunction(
File "/home/nankaingy/data/anaconda3/envs/actor/lib/python3.8/site-packages/OpenGL/platform/baseplatform.py", line 148, in constructFunction
if (not is_core) and not self.checkExtension( extension ):
File "/home/nankaingy/data/anaconda3/envs/actor/lib/python3.8/site-packages/OpenGL/platform/baseplatform.py", line 270, in checkExtension
result = extensions.ExtensionQuerier.hasExtension( name )
File "/home/nankaingy/data/anaconda3/envs/actor/lib/python3.8/site-packages/OpenGL/extensions.py", line 98, in hasExtension
result = registered( specifier )
File "/home/nankaingy/data/anaconda3/envs/actor/lib/python3.8/site-packages/OpenGL/extensions.py", line 105, in call
if not specifier.startswith( self.prefix ):
TypeError: startswith first arg must be bytes or a tuple of bytes, not str

Anyone can help?

Thanks in advance!