snap-research / articulated-animation Goto Github PK

Hey @AliaksandrSiarohin
Thanks a lot for sharing such awesome work. All the best for the conference!

I had a hard time replicating the work and was wondering if you could share some inputs here

• Parameters for selecting source image compatible with driving video
  • Body Visibility?
  • Initial pose?
  • Gender/Age
• For head re-enactment (similar to First order motion model), is the procedure same as before?

I find that the demo does not use the background predictor. Could you please to release the related demo using bg predictor? How does this scheme perform in the vox dataset? Looking forward to your reply~

Hi, Aliaksandr. Your code is really nice.
However, I find a small mistake here from L118 to L120:

Shouldn't it be result = (distances ** 2).sum(-1)?
With which change, the code will be more consistent with:
https://github.com/tomasjakab/imm/blob/0fee6b24466a5657d66099694f98036c3279b245/imm/utils/tps_sampler.py#L138

@AliaksandrSiarohin
Hi, I have two simple questions about the vox256 model in your provided checkpoints.

• What is the resolution of the data for training vox256, 256x256, or 384x384?
• This model is trained from which version of the VoxCeleb dataset? V1 or V2?

any idea how to fix?

When I run your requirements install I get an error on torch, can a newer version be used? If not, how can I get the 1.4.0 version you require?

PS C:\Users\gcorson\Documents\GitHub\articulated-animation> pip install -r requirements.txt
Collecting imageio==2.3.0
Using cached imageio-2.3.0-py2.py3-none-any.whl (3.3 MB)
Collecting matplotlib==2.2.2
Using cached matplotlib-2.2.2-cp37-cp37m-win_amd64.whl (8.7 MB)
Collecting numpy==1.15.0
Using cached numpy-1.15.0-cp37-none-win_amd64.whl (13.5 MB)
Collecting pandas==0.23.4
Using cached pandas-0.23.4-cp37-cp37m-win_amd64.whl (7.9 MB)
Collecting Pillow==5.2.0
Using cached Pillow-5.2.0-cp37-cp37m-win_amd64.whl (1.6 MB)
Collecting PyYAML==5.1
Using cached PyYAML-5.1-cp37-cp37m-win_amd64.whl (215 kB)
Collecting scikit-image==0.14.0
Using cached scikit_image-0.14.0-cp37-none-win_amd64.whl (24.6 MB)
Collecting scikit-learn==0.19.2
Using cached scikit_learn-0.19.2-cp37-cp37m-win_amd64.whl (4.4 MB)
Collecting scipy==1.1.0
Using cached scipy-1.1.0-cp37-none-win_amd64.whl (30.9 MB)
ERROR: Could not find a version that satisfies the requirement torch==1.4.0 (from versions: 0.1.2, 0.1.2.post1, 0.1.2.post2, 1.7.0, 1.7.1, 1.8.0, 1.8.1)
ERROR: No matching distribution found for torch==1.4.0

Hi @AliaksandrSiarohin
When I downloaded the TED dataset , I found that two video was downloaded failed with "Can not load video xDKth-qS8Jk, broken link" and "Can not load video _QdPW8JrYzQ, broken link". I have tried other download methods and got the same result , and I met same question with voxceleb1 dataset， which has more broken videos. So I want to ask how can I solve this problem?
Thanks！

Hi @AliaksandrSiarohin,

May I ask a question about the warping result like this:

It seems the moved region is filled in the warping image (the third column) based on the estimated optical flow. I try to use the off-the-self model to predict the optical flow, however, the warping result have the issue about double region like this:

I would appreciate it if you can give me any hint to solve this problem.

When I run 'git clone https://github.com/snap-research/articulated-animation/'
an error occurs:
Error downloading object: checkpoints/mgif256.pth (58b796e): Smudge error: Error downloading checkpoints/mgif256.pth (58b796e31f763ccfbb240f959c9d92c2afe8a6e57da303e07d8fabd1f2921c68): batch response: This repository is over its data quota. Account responsible for LFS bandwidth should purchase more data packs to restore access.

Then i try to run 'git lfs pull'
The following error occurs:
batch response: This repository is over its data quota. Account responsible for LFS bandwidth should purchase more data packs to restore access.
error: failed to fetch some objects from 'https://github.com/snap-research/articulated-animation.git/info/lfs'

Is there anything wrong with that?
Thanks

Hello. Again, great work! I'm a little confused about the training part. You say:

To train a model run:

CUDA_VISIBLE_DEVICES=0 python run.py --config config/dataset_name.yaml --device_ids 0

Then there's this:

Then to train Animation via disentaglement (AVD) use:

CUDA_VISIBLE_DEVICES=0 python run.py --checkpoint log/{folder}/cpk.pth --config config/dataset_name.yaml --device_ids 0 --mode train_avd

Do I train the model fully (100 epoches) before moving onto the AVD training, or do I train exactly one checkpoint first, then start the AVD training? Thanks!

Hi @AliaksandrSiarohin
Is it possible to attain output of 512x512, by retraining using vox256 checkpoint file ?
Please suggest what can be done for 512x512 output.

Thanks

no discriminator for training?

In your paper, you just said it was cropped by detectron2. Since I need to test the ted384.pth on my own dataset, I need to know your crop method detailly. Thank you.

Hi! I trained the model on jhmdb dataset and found some heatmaps were poorly generated.

As shown above, the heatmaps of the video sequence didn't capture the expected regions(e.g. arms, legs). Is there a way to modify the heatmaps? Or can I use a pre-trained semantic segmentation model for heatmap generation? Thank you.

Screenshot:

Should point to: https://colab.research.google.com/github/snap-research/articulated-animation/blob/main/demo.ipynb

It seems the results u and v from SVD given by
region_params['u'] = u
region_params['d'] = d
from region_generator.py
does not participated with the training process?

How can we view the pixel wise flow in Figure 2? The output is of shape batch x 64 x 64 x 2

First of all, thank you very much for providing the code, but I have encountered some small problems in the process of retraining, so I would like to ask you how to deal with it. Questions to consult are as follows:
1、why reconstruction mode and train model with almost same L1 loss value?
2、Using the 768 size TED dataset, it is normal that some parts with more detailed information, such as hands and faces, are not recovered too well. If the current situation occurs, can you help to provide some solutions?
3. When the motion trend is obvious, the optical flow map is not very accurate.
4. Are there any precautions that need to take in preparing new dataset?
The above are all my questions at present. Looking forward to your reply

i get some errors after i installed the dependencies. To try the animation demo.

ModuleNotFoundError: No module named 'numpy.core._multiarray_umath'.

ImportError: Matplotlib qt-based backends require an external PyQt4, PyQt5,

ImportError: cannot import name '_validate_lengths' from 'numpy.lib.arraypad'

The following command solved my problems
pip3 install numpy==1.16.0 scikit-image==0.14.2 pyside2==5.14.0

My system
Python 3.8.10 on Linux Mint 20.2 Uma \n \l

Hope it helps someone. Many greetings

Getting the error:

Error downloading object: checkpoints/mgif256.pth (58b796e): Smudge error: Error downloading checkpoints/mgif256.pth (58b796e31f763ccfbb240f959c9d92c2afe8a6e57da303e07d8fabd1f2921c68): batch response: This repository is over its data quota. Account responsible for LFS bandwidth should purchase more data packs to restore access.

I try to run the demo script for test, but an error occured: _pickle.UnpicklingError: invalid load key, 'v'.
I'm new to these and I have no idea how to solve it.
The command I used: python demo.py --config config/ted384.yaml --driving_video sup-mat/driving.mp4 --source_image sup-mat/source.png --checkpoint checkpoints/ted384.pth
Thank you so much for your reply : )

First, thank you for your valuable research.
I would like to ask for your opinion on how to approach dataset to train like DAVIS-dataset(https://davischallenge.org/).

The expected problems are as follows.

1. DAVIS has multiple categories.
   DAVIS Dataset has a variety of data, including dancers, boxers, elephants, cars, horses, and about 80 to 90 categories include one video. The algorithm seems to have strengths in a single category, is there a good way to deal with various categories? Should we just collect more data and integrate similar categories as much as possible to learn only one category?

2. Background is very dynamic.
   Due to the nature of dataset, the background is so chaotic that it is difficult to separate objects and background properly (from a learning perspective). To solve this problem, is it right to mask the video only for the data that contains segmentation mask(annotation or label) and forcibly remove the background to create a new dataset that contains only the motion of the object?

I really like the author's thesis and execution code, so I want to use it for my project. Thank you.

Thanks for sharing your nice work!

I am trying to retrain your model on TED dataset, and run the command:

CUDA_VISIBLE_DEVICES=0,1 python run.py --config config/ted384.yaml --device_ids 0,1

With

Batch_size = 8, num_epochs: 100, num_repeats: 150, input: image with png format.

It seems the training process will take more than 7 days. May I know how long did you take? Thanks.

Thanks for sharing your nice work!
I meet a problem when downloading TED dataset. I get:

/home/yzhang4/anaconda3/envs/motion/lib/python3.6/importlib/_bootstrap.py:205: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88
  return f(*args, **kwds)
/home/yzhang4/anaconda3/envs/motion/lib/python3.6/importlib/_bootstrap.py:205: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88
  return f(*args, **kwds)
0it [00:00, ?it/s]
multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
  File "/home/yzhang4/anaconda3/envs/motion/lib/python3.6/multiprocessing/pool.py", line 119, in worker
    result = (True, func(*args, **kwds))
  File "load_videos.py", line 32, in run
    download(video_id.split('#')[0], args)
  File "load_videos.py", line 25, in download
    video_path], stdout=DEVNULL, stderr=DEVNULL)
  File "/home/yzhang4/anaconda3/envs/motion/lib/python3.6/subprocess.py", line 267, in call
    with Popen(*popenargs, **kwargs) as p:
  File "/home/yzhang4/anaconda3/envs/motion/lib/python3.6/subprocess.py", line 707, in __init__
    restore_signals, start_new_session)
  File "/home/yzhang4/anaconda3/envs/motion/lib/python3.6/subprocess.py", line 1326, in _execute_child
    raise child_exception_type(errno_num, err_msg)
PermissionError: [Errno 13] Permission denied
"""
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
  File "load_videos.py", line 103, in <module>
    for chunks_data in tqdm(pool.imap_unordered(run, zip(video_ids, args_list))):
  File "/home/yzhang4/anaconda3/envs/motion/lib/python3.6/site-packages/tqdm/std.py", line 1178, in __iter__
    for obj in iterable:
  File "/home/yzhang4/anaconda3/envs/motion/lib/python3.6/multiprocessing/pool.py", line 699, in next
    raise value
PermissionError: [Errno 13] Permission denied

Could you please give me some advice to solve this problem?

Hi, great work! I have some questions about the training part.
1、Are the models in your provided checkpoints trained by the mode of AVD training(train_avd)?
2、Dose bg_predictor not work when AVD training?

Hello Alexandre,

Thanks a lot for all the amazing works and sharing them with the community. Let me go straight to my question. In your paper, you have provided a comparison between your model and First Order Motion. I'm just wondering if it is possible for you to put the pretrained model (checkpoint) of the FOM model here or in its corresponding github page.

Thank you in advance,

while running
python run.py --config config/ted384.yaml --device_ids 0,1
I am getting the following error.

A similar error has been pointed here AliaksandrSiarohin/first-order-model#217
I see the memtest is already set to False in the codebase. Not sure why this error.

On running:
python load_videos.py --metadata ../data/ted384-metadata.csv --format .mp4 --out_folder ../data/TED384-v2 --image_shape 384,384

I get:

0it [00:00, ?it/s]Unknown encoder 'libx264'
multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
File "/home2/neelabh/miniconda3/envs/py36/lib/python3.6/site-packages/imageio/plugins/ffmpeg.py", line 661, in _append_data
self._proc.stdin.write(im.tostring())
BrokenPipeError: [Errno 32] Broken pipe

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home2/neelabh/miniconda3/envs/py36/lib/python3.6/multiprocessing/pool.py", line 119, in worker
result = (True, func(*args, **kwds))
File "load_videos.py", line 72, in run
save(os.path.join(args.out_folder, partition, path), entry['frames'], args.format)
File "/home2/neelabh/articulated-animation/video-preprocessing/util.py", line 118, in save
imageio.mimsave(path, frames)
File "/home2/neelabh/miniconda3/envs/py36/lib/python3.6/site-packages/imageio/core/functions.py", line 341, in mimwrite
writer.append_data(im)
File "/home2/neelabh/miniconda3/envs/py36/lib/python3.6/site-packages/imageio/core/format.py", line 492, in append_data
return self._append_data(im, total_meta)
File "/home2/neelabh/miniconda3/envs/py36/lib/python3.6/site-packages/imageio/plugins/ffmpeg.py", line 666, in _append_data
raise IOError(msg)
OSError: [Errno 32] Broken pipe

FFMPEG COMMAND:
/home2/neelabh/miniconda3/envs/py36/bin/ffmpeg -y -f rawvideo -vcodec rawvideo -s 384x384 -pix_fmt rgb24 -r 10.00 -i - -an -vcodec libx264 -pix_fmt yuv420p -crf 25 -v warning /home2/neelabh/articulated-animation/data/TED384-v2/test/yMWlkJAqKYU#005708#005838.mp4

FFMPEG STDERR OUTPUT:

"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "load_videos.py", line 100, in
for chunks_data in tqdm(pool.imap_unordered(run, zip(video_ids, args_list))):
File "/home2/neelabh/miniconda3/envs/py36/lib/python3.6/site-packages/tqdm/_tqdm.py", line 931, in iter
for obj in iterable:
File "/home2/neelabh/miniconda3/envs/py36/lib/python3.6/multiprocessing/pool.py", line 735, in next
raise value
OSError: [Errno 32] Broken pipe

FFMPEG COMMAND:
/home2/neelabh/miniconda3/envs/py36/bin/ffmpeg -y -f rawvideo -vcodec rawvideo -s 384x384 -pix_fmt rgb24 -r 10.00 -i - -an -vcodec libx264 -pix_fmt yuv420p -crf 25 -v warning /home2/neelabh/articulated-animation/data/TED384-v2/test/yMWlkJAqKYU#005708#005838.mp4

FFMPEG STDERR OUTPUT:

According to this answer on stack overflow, it may be because the request is getting blocked or taking too long.

When I tried to download the Ted dataset, it seemed that all the links failed. I don't know how to fix it.
In addition, I want to figure out the format of the dataset in the file. I find that the generalization ability of the model is limited for the data collected in the wild, even if I try to use all the pre trained models
Thanks!

/home/liuxinqi/anaconda3/envs/art-ani/lib/python3.7/site-packages/scipy/init.py:140: UserWarning: A NumPy version >=1.16.5 and <1.23.0 is required for this version of SciPy (detected version 1.15.0)
UserWarning)
0it [00:00, ?it/s]Can not load video iMFJef3xnmg, broken link
1it [00:27, 27.56s/it]Can not load video Cetg4gu0oQQ, broken link
Can not load video 360bU-vBJOI, broken link
Can not load video xHHb7R3kx40, broken link
Can not load video LnJwH_PZXnM, broken link
5it [00:27, 5.53s/it]Can not load video ClfBxWPkBKU, broken link
Can not load video cK74vhqzeeQ, broken link
Can not load video 1g-1_Y3fvUg, broken link
8it [00:28, 3.59s/it]Can not load video lyR-K2CZIHQ, broken link
9it [00:29, 3.25s/it]Can not load video 95ovIJ3dsNk, broken link
Can not load video kBBmVezBUkg, broken link
Can not load video iFTWM7HV2UI, broken link
12it [00:29, 2.44s/it]Can not load video WfTZ5iIUn4s, broken link
Can not load video t0Cr64zCc38, broken link
14it [00:29, 2.13s/it]Can not load video kyaiTGmwxnU, broken link
Can not load video 9alL95G293s, broken link
16it [00:30, 1.90s/it]Can not load video -nKdufEaL8k, broken link
17it [00:30, 1.82s/it]Can not load video oEIYHTlbeLA, broken link
18it [00:31, 1.73s/it]Can not load video v9EKV2nSU8w, broken link
19it [00:31, 1.65s/it]Can not load video 51k3UASQE5E, broken link
Can not load video VM6HZqQKhok, broken link
Can not load video 51k3UASQE5E, broken link
22it [00:31, 1.44s/it]Can not load video HI7zfpitZpo, broken link
23it [00:31, 1.38s/it]Can not load video FDhlOovaGrI, broken link
24it [00:32, 1.36s/it]Can not load video 08z_xW-szwM, broken link
Can not load video fO2htapfNhA, broken link
26it [00:32, 1.26s/it]Can not load video idfv7Lw4Y_s, broken link
27it [00:32, 1.22s/it]Can not load video I3BJVaioX_k, broken link
28it [00:33, 1.18s/it]Can not load video MB5IX-np5fE, broken link
29it [00:33, 1.15s/it]Can not load video FqrLUtIFVjs, broken link
30it [00:33, 1.12s/it]Can not load video 13zN4-MVM9g, broken link
31it [00:33, 1.08s/it]Can not load video iFTWM7HV2UI, broken link
32it [00:34, 1.07s/it]Can not load video HiwJ0hNl1Fw, broken link
33it [00:34, 1.04s/it]Can not load video 1AT5klu_yAQ, broken link
34it [00:34, 1.02s/it]Can not load video TLZ6W-Nqv1I, broken link
Can not load video iMFJef3xnmg, broken link
36it [00:34, 1.03it/s]Can not load video pxEcvU0Vp_M, broken link
Can not load video kmbui1xF8DE, broken link
38it [00:35, 1.07it/s]Can not load video 2VBkDNzeRZM, broken link
Can not load video yNhu0MG_2MA, broken link
40it [00:36, 1.11it/s]Can not load video VJoQj00RZHg, broken link
41it [00:36, 1.13it/s]Can not load video SF9qq6vQ3Pg, broken link
multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
File "/home/liuxinqi/anaconda3/envs/art-ani/lib/python3.7/multiprocessing/pool.py", line 121, in worker
result = (True, func(*args, **kwds))
File "load_videos.py", line 72, in run
save(os.path.join(args.out_folder, partition, path), entry['frames'], args.format)
File "/home/liuxinqi/disk/study/articulated-animation-main/video-preprocessing/util.py", line 118, in save
imageio.mimsave(path, frames)
File "/home/liuxinqi/anaconda3/envs/art-ani/lib/python3.7/site-packages/imageio/core/functions.py", line 347, in mimwrite
raise RuntimeError('Zero images were written.')
RuntimeError: Zero images were written.
"""

Thanks for your nice work!
I have a question about the evaulation: are the source and diriving frames from the same video for quantitative evaulation listed in Table 2?

I noticed there are no examples from VoxCeleb in the paper or the code. Also no sufficient information or data so one can replicate the experiments. Can you please share?

“ an arm moving across the body, is not well modeled, producing unconvincing animations”
the result is not good of face and arm in the pictures, how to improve?


thanks!

On running:
python load_videos.py --metadata ../data/ted384-metadata.csv --format .mp4 --out_folder ../data/TED384-v2 --workers 8 --image_shape 384,384

I get:

Traceback (most recent call last):
File "load_videos.py", line 96, in
df = pd.read_csv(args.metadata)
File "/home2/neelabh/miniconda3/envs/py36/lib/python3.6/site-packages/pandas/io/parsers.py", line 678, in parser_f
return _read(filepath_or_buffer, kwds)
File "/home2/neelabh/miniconda3/envs/py36/lib/python3.6/site-packages/pandas/io/parsers.py", line 440, in _read
parser = TextFileReader(filepath_or_buffer, **kwds)
File "/home2/neelabh/miniconda3/envs/py36/lib/python3.6/site-packages/pandas/io/parsers.py", line 787, in init
self._make_engine(self.engine)
File "/home2/neelabh/miniconda3/envs/py36/lib/python3.6/site-packages/pandas/io/parsers.py", line 1014, in _make_engine
self._engine = CParserWrapper(self.f, **self.options)
File "/home2/neelabh/miniconda3/envs/py36/lib/python3.6/site-packages/pandas/io/parsers.py", line 1708, in init
self._reader = parsers.TextReader(src, **kwds)
File "pandas/_libs/parsers.pyx", line 384, in pandas._libs.parsers.TextReader.cinit
File "pandas/_libs/parsers.pyx", line 695, in pandas._libs.parsers.TextReader._setup_parser_source
FileNotFoundError: File b'../data/ted384-metadata.csv' does not exist

Is this as simple as setting bg_type flag in the config file to zero?

Thanks for you sharing code, This is a very great project. But I run, appear error, as follows:

Traceback (most recent call last):
  File "demo.py", line 134, in <module>
    main(parser.parse_args())
  File "demo.py", line 113, in main
    checkpoint_path=opt.checkpoint, cpu=opt.cpu)
  File "demo.py", line 59, in load_checkpoints
    checkpoint = torch.load(checkpoint_path)
  File "/home/zhaopeng/anaconda3/envs/articulatedanimation/lib/python3.6/site-packages/torch/serialization.py", line 529, in load
    return _legacy_load(opened_file, map_location, pickle_module, **pickle_load_args)
  File "/home/zhaopeng/anaconda3/envs/articulatedanimation/lib/python3.6/site-packages/torch/serialization.py", line 692, in _legacy_load
    magic_number = pickle_module.load(f, **pickle_load_args)
_pickle.UnpicklingError: invalid load key, 'v'.

my command, as follows:
CUDA_VISIBLE_DEVICES='0' python demo.py --config config/ted384.yaml --driving_video nv.mp4 --source_image nv.png --checkpoint checkpoints/ted384.pth --mode avd

my environment, as follows:

torch 1.4.0
torchvision 0.5.0

thers is a little different from requirement.txt, your torchvision is 0.2.1.

Could you help me, thanks.

The following error is received

Use git lfs logs last to view the log.
batch response: This repository is over its data quota. Account responsible for LFS bandwidth should purchase more data packs to restore access.
Failed to fetch some objects from 'https://github.com/snap-research/articulated-animation.git/info/lfs'

Could you share taichi512.yaml and taichi512-metadata.csv if available?

Hi, thanks for sharing the code. And I notice that in the paper, TaiChi dataset of resolution 512*512 is used for experiments, and I want to make a reproducement of that. Could you provide the TaiChiHD-512 dataset and taichi-512.yaml ?

Hi Team ,

I have applied your model on mgif dataset sample but it is not working as expected and give distorted results.
I used the sample source and driving video that you have shown to the papers but your results are quite good!

What will be possible reasons for this?

Hello. I have read your paper and your code carefully. I think this is a very high-quality job.
But when I train the model, I found that some regions are not oval (like the hair in TED, or the eyes in VOX). Here are some examples.

I check the implementation of the flow predictor. It generates the sparse motion by rendering a gaussian heatmap according to the region params.
In my experience, the Gaussian distribution generates elliptical regions.
Considering some regions are not oval, can the sparse motion accurate enough? Or is there some other mechanism to handle this situation? Do you think this could be a direction for improvement in the future?

I have a small question related to the revert_axis_swap operation, I notice you flip the sign of the affine matrix when the first element is negative, what are the benefits of this?

Really important. Just asking for science.

Hi, I use the provided checkpoint of MGIF dataset to run evaluation code for the "mode" reconstruction, and get the L1 metric of 0.027 while the paper report the 0.0206. There is a big difference between them. So I doubt that if I make sth. wrong. I use the pytorch version 1.4.0.

Cannot download the checkpoints from git lfs.Thanks

Couldn't find 'ted384-metadata.csv' in './data' folder.

Instead, there is a file 'ted-metadata.csv ''that has all 0s for 'start' and 'end' entries (but start/end frame numbers are included in video_id).

Is it OK to modify 'load_videos.py' to support this configuration in ted-metadata.csv?

But, I wonder whether ted-metadata.csv is the same as ted384-metadata.csv except start/end frames issue.

Hey, i'm really interested in using this on bitmojis. But none of the checkpoints seem to be generating satisfactory results. Any advice for the best way to leverage human driving videos for bitmoji souce images ?