wpeebles / gangealing Goto Github PK

question one: on training, the loss is always 0
question two: "transformed_sample_0******.png" become one-color and only 6kb after 105000 iters,

why did I fail to train the model?
Thanks!

I didn't change the code. The weights was auto download.

script:

CUDA_VISIBLE_DEVICES=7 python train.py \
--ckpt cat --load_G_only --padding_mode border --vis_every 5000 --ckpt_every 50000 \
--iter 1500000 --tv_weight 1000 --loss_fn vgg_ssl --exp-name debug

out:

Setting up [baseline] perceptual loss: trunk [vgg], v[0.1], spatial [off]
Loading VGG with pretrained=False
Loaded custom VGG weights.
Loading model from cat
Only G_EMA has been loaded from checkpoint. Other nets are random!
Fitting PCA...
Learning Rate Cycles: [149999, 187499, 262499, 412499, 712499, 1312499]
perceptual loss: 0.1561; tv loss: 0.000000; identity loss: 0.0000; psi: 1.0000:

Hi,
I try to run

CUDA_VISIBLE_DEVICES=7 python train.py  --ckpt cat --load_G_only --padding_mode border --vis_every 5000 --ckpt_every 50000  --iter 1500000 --tv_weight 2500 --loss_fn lpips --exp-name lsun_cats_lpips

and get several different errors when compiling fused_bias_act_kernel.cu according to the PyTorch and Cuda version I use. I have try pytorch 1.2.1, 1.4.0, 1.8.0 with cuda 10.0, 10.1 , 11.1.

Here is an example error with pytorch 1.8.0, cuda 11.1:

Traceback (most recent call last):
  File "train.py", line 13, in <module>
    from models import Generator, get_stn, DirectionInterpolator, PCA
  File "/home/username/nvidia_code/gangealing/models/__init__.py", line 2, in <module>
    from models.stylegan2.networks import Generator
  File "/home/username/nvidia_code/gangealing/models/stylegan2/networks.py", line 7, in <module>
    from models.stylegan2.op import FusedLeakyReLU, fused_leaky_relu, upfirdn2d, conv2d_gradfix
  File "/home/username/nvidia_code/gangealing/models/stylegan2/op/__init__.py", line 1, in <module>
    from .fused_act import FusedLeakyReLU, fused_leaky_relu
  File "/home/username/nvidia_code/gangealing/models/stylegan2/op/fused_act.py", line 15, in <module>
    os.path.join(module_path, "fused_bias_act_kernel.cu"),
  File "/home/username/anaconda2/envs/gg/lib/python3.7/site-packages/torch/utils/cpp_extension.py", line 1091, in load
    keep_intermediates=keep_intermediates)
  File "/home/username/anaconda2/envs/gg/lib/python3.7/site-packages/torch/utils/cpp_extension.py", line 1302, in _jit_compile
    is_standalone=is_standalone)
  File "/home/username/anaconda2/envs/gg/lib/python3.7/site-packages/torch/utils/cpp_extension.py", line 1407, in _write_ninja_file_and_build_library
    error_prefix=f"Error building extension '{name}'")
  File "/home/username/anaconda2/envs/gg/lib/python3.7/site-packages/torch/utils/cpp_extension.py", line 1683, in _run_ninja_build
    raise RuntimeError(message) from e
RuntimeError: Error building extension 'fused': [1/2] /cm/shared/apps/cuda100/10.0.130/bin/nvcc --generate-dependencies-with-compile --dependency-output fused_bias_act_kernel.cuda.o.d -D
TORCH_EXTENSION_NAME=fused -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -isystem /home/zitianch
en/anaconda2/envs/gg/lib/python3.7/site-packages/torch/include -isystem /home/username/anaconda2/envs/gg/lib/python3.7/site-packages/torch/include/torch/csrc/api/include -isystem /home
/username/anaconda2/envs/gg/lib/python3.7/site-packages/torch/include/TH -isystem /home/username/anaconda2/envs/gg/lib/python3.7/site-packages/torch/include/THC -isystem /cm/shared/a
pps/cuda100/10.0.130/include -isystem /home/username/anaconda2/envs/gg/include/python3.7m -D_GLIBCXX_USE_CXX11_ABI=0 -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUD
A_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr -gencode=arch=compute_75,code=compute_75 -gencode=arch=compute_75,code=sm_75 --compiler-options '-fPIC'
 -std=c++14 -c /home/username/nvidia_code/gangealing/models/stylegan2/op/fused_bias_act_kernel.cu -o fused_bias_act_kernel.cuda.o
FAILED: fused_bias_act_kernel.cuda.o
/cm/shared/apps/cuda100/10.0.130/bin/nvcc --generate-dependencies-with-compile --dependency-output fused_bias_act_kernel.cuda.o.d -DTORCH_EXTENSION_NAME=fused -DTORCH_API_INCLUDE_EXTENSI
ON_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -isystem /home/username/anaconda2/envs/gg/lib/python3.7/site-packages/torch
/include -isystem /home/username/anaconda2/envs/gg/lib/python3.7/site-packages/torch/include/torch/csrc/api/include -isystem /home/username/anaconda2/envs/gg/lib/python3.7/site-packages/torch/include/TH -isystem /home/username/anaconda2/envs/gg/lib/python3.7/site-packages/torch/include/THC -isystem /cm/shared/apps/cuda100/10.0.130/include -isystem /home/username
/anaconda2/envs/gg/include/python3.7m -D_GLIBCXX_USE_CXX11_ABI=0 -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATOR
S__ --expt-relaxed-constexpr -gencode=arch=compute_75,code=compute_75 -gencode=arch=compute_75,code=sm_75 --compiler-options '-fPIC' -std=c++14 -c /home/username/nvidia_code/gangealing
/models/stylegan2/op/fused_bias_act_kernel.cu -o fused_bias_act_kernel.cuda.o
nvcc fatal   : Unknown option '-generate-dependencies-with-compile'
ninja: build stopped: subcommand failed.

Any clue to solve this? I already check the stylegan2 repository but found nothing. Can you tell me what pytorch and cuda version I should use?

hello wpeebles,
i try to train the styleGan net with my own datasets,and i used a 2080Ti 12G GPU batchsize=4 to train 4 days,
it returned this,and I want to know if I didn't have enough training time or if I failed.

I'm getting a ImportError: Cannot compile splatting CUDA library.

Loading extension module _splat...
0%| | 0/1 [00:00<?, ?it/s]

ImportError Traceback (most recent call last)
/content/gangealing/utils/splat2d_cuda/functional.py in _import_splat()
17
---> 18 _splat = load_extension(
19 '_splat',

13 frames
ImportError: /root/.cache/torch_extensions/py310_cu118/_splat/_splat.so: cannot open shared object file: No such file or directory

During handling of the above exception, another exception occurred:

ImportError Traceback (most recent call last)
/content/gangealing/utils/splat2d_cuda/functional.py in _import_splat()
24 )
25 except ImportError:
---> 26 raise ImportError('Cannot compile splatting CUDA library.')
27
28 return _splat

ImportError: Cannot compile splatting CUDA library.

Hello: thanks for the awsome work and consistently helping with the issues!

I'm running visualization on a model that I trained on my own dataset. I was able to reproduce the dense corresponded mix reality on video and propagation to each individual frame images with a mask that I created my own.

I wonder is there any way that I can get the mask position / coordination in image from the model after I applied the dense tracking?

thanks in advance!

Let's say I have a dataset which I've preprocessed with gangealing and I have saved the dense_correspondences.py in a file.

How can I map the congealed frame pixels back to the original image using the dense_correspondences?

if I want to train the LPIPS(Perceptual Losses) myself, where could I found the paper？

is this project: https://github.com/richzhang/PerceptualSimilarity

Thanks！

I run the following script:

torchrun --nproc_per_node=4 train.py \
--ckpt cat --load_G_only --padding_mode border --vis_every 5000 --ckpt_every 50000 \
--iter 1500000 --tv_weight 1000 --loss_fn vgg_ssl --exp-name lsun_cats --batch 10

and find the loss is going up and the transformed image learn almost nothing.
Also, it take 1.85s/iter and need 1500000 iter which cost ~220 hour. Is that normal?

Hello, I noticed that on this GitHub page, substantial information has been given to run the unimodal STNs. I tried running the multi-modal STN (I used http://efrosgans.eecs.berkeley.edu/gangealing/pretrained/horse.pt for the multi-modal horse model and I attempted to use cluster 0 of the horse clustering model) by following your 'using-pre-trained-clustering-models' (https://github.com/wpeebles/gangealing#using-pre-trained-clustering-models) section by adding the extra arguments. my command is as such:

!torchrun --nproc_per_node=1 applications/mixed_reality.py --ckpt horse --objects --label_path assets/objects/horse_cluster0/horse_cluster0_saddle.png --average_path assets/averages/horse_cluster0.png --num_heads 4 --sigma 0.3 --opacity 1 --cluster 0 --real_size 1024 --resolution 8192 --real_data_path data/final_horse

but I am getting the following error on Google Colab:

Also, I tried to run the multi-modal STN via the google colab link provided in this github page. These are the arguments i used:

args.real_size = int(video_size)
args.real_data_path = video_path
args.fps = fps
args.batch = batch_size
args.blend_alg = blend_alg
args.transform = ['similarity', 'flow']
args.flow_size = 128
args.stn_channel_multiplier = 0.5
args.num_heads = 4
args.distributed = False # Colab only uses 1 GPU
args.clustering = True
args.cluster = [0] #to prevent errors. try using 'args.cluster = 0' to replicate another error i got.
args.objects = True
args.no_flip_inference = not use_flipping
args.save_frames = memory_efficient_but_slower
args.overlay_congealed = False
args.ckpt = save_model
args.override = False
args.save_correspondences = False
args.out = 'visuals'
args.average_path = f'assets/averages/horse_cluster0.png'

and this is the error i got from using the multi-modal STN via the google colab link:

I'm not sure what I'm doing wrong - Could I have some assistance or guidance regarding this matter? Thank you in advance.

Hi. In prepare_data.py, in line 22 you state that "When an image is mirrored, any key points with left/right distinction need to be swapped. These are the permutations of key point indices that accomplishes this:". How did you calculate these permutations, and how could I calculate them for another class of the Spair-71k dataset, e.g. person?
Thank you in advance :)

I tried to run the script for CUB data preparation -
python prepare_data.py --cub_acsm --out data/cub_val --size 256

I'm getting:

FileNotFoundError: [Errno 2] No such file or directory: 'data/val_cub_cleaned_new.mat'

How can I download this file?

hello,i have something problem! wuuuu

i run this command :
python -m torch.distributed.launch --nproc_per_node=8 train.py --ckpt cat --load_G_only --padding_mode border --vis_every 5000 --ckpt_every 50000 --iter 1500000 --tv_weight 1000 --loss_fn vgg_ssl --exp-name lsun_cats

thanks！

A official dockerfile would be nice for use in servers and the likes

Hi! I've seem many other post mentioned batch size is essential here to keep the training stable. However I do only have one GPU which only allows me to use batch size under 8. I wonder is there any ways that I can reduce the number of RGB channel of the model to 1? The task / dataset I'm working on is relatively simple and can basically ignore the colors.

Thanks in advance!

Hi, would you be interested in adding gangealing to Hugging Face as a Gradio Web Demo for CVPR 2022? The Hub offers free hosting, and it would make your work more accessible and visible to the rest of the ML community. Models/datasets/spaces(web demos) can be added to a user account or organization similar to github.

more info see CVPR organization on Hugging Face: https://huggingface.co/CVPR

here is a example Gradio Demo for the CVPR org: https://huggingface.co/spaces/CVPR/ml-talking-face

and here is a guide for adding web demo to the organization: https://huggingface.co/blog/gradio-spaces

Please let us know if you would be interested and if you have any questions, we can also help with the technical implementation.

Hello, thanks for your excellent work!

As shown in the figure, the position of keypoints in image B is random (depends on the SPair dataset). However, your model will transform input image(here is image A) to a modification of average image (almost the front). So, how do your calculate the PCK between gt image B and the transformed image?

Hi, your work is really cool. I wonder can it serve as an optical flow estimator? If yes, could you please show me the demo code? By the way, what are its advantages and disadvantages when compared with the conventional optical flow networks, e.g., PWCNet ?

Hi, I tried videos with size 16:9 (1280*720) and results is cropped with size 1:1, can I keep the original size of the result video?

python applications/vis_correspondence.py --ckpt cat --real_data_path data/lsun_cats --vis_in_stages --real_size 512 --output_resolution 512 --resolution 512 --label_path assets/masks/cat_mask.png --dset_indices 1922 2363 8558 7401 9750 7432 2105 53 1946

Traceback (most recent call last):
File "applications/vis_correspondence.py", line 22, in
from datasets import MultiResolutionDataset, img_dataloader
ModuleNotFoundError: No module named 'datasets'

err report:

1. OS version : windows server 2019
2. The envs had activate,and I have installed all the installation steps and required dependencies with no error warnings.
3. err detailed :
   (gg) C:\Users\Administrator\gangealing>torchrun --nproc_per_node=1 applications/mixed_reality.py --ckpt cat --objects --label_path assets/objects/cat/cat_cartoon.png --sigma 0.3 --opacity 1 --real_size 512 --resolution 8192 --real_data_path data\video_frames\white_cat-PNG --no_flip_inferencee
   failed to create process.

I don't know why this error occurred,and how to fix it.

Would be awesome to try out this project on google colab?

Also, would this kind of inference be possible on device (say on a mediatek dimensity 1200 chip ? )

Hello!

I was wondering if it was possible to add masks on the upper forehead/lower chin. The congealed images are are cut off at both the forehead and the chin (See here). I was hoping to apply a mask on the full face including these regions.

Thanks and very cool work!

how to produce a single pic, not a video

When I tried mixed reality demo and add moustache on celeba model, I ran

torchrun --nproc_per_node=4 applications/mixed_reality.py --ckpt celeba --objects --label_path assets/objects/celeba/celeba_moustache.png --sigma 0.3 --opacity 1 --real_size 1024 --resolution 2048 --real_data_path path_to_my_video --no_flip_inference

where I use 4 Nvidia 3090 GPUs and set --resolution = 2048 because of the memory limitation. The result makes sense but has some weird jitter as:

I checked the frames of the result and the images are normal with moustache. How do I fix this? Is there any incorrect parameters of my running?

In this project's code, the sylegan is likely based on https://github.com/rosinality/stylegan2-pytorch

But I found no args setting in https://github.com/rosinality/stylegan2-pytorch.

I have try to set args refer to https://github.com/NVlabs/stylegan2, but the result is not good.

Could you share your training scripts for the stylegan2?

Hi, I try to run any command and will get this error:

No CUDA runtime is found, using CUDA_HOME='/cm/shared/apps/cuda100/10.0.130'
Traceback (most recent call last):
  File "applications/propagate_to_images.py", line 20, in <module>
    from applications import base_eval_argparse, load_stn, determine_flips
  File "/home/user/nvidia_code/gangealing/applications/__init__.py", line 3, in <module>
    from models import get_stn, ResnetClassifier
  File "/home/user/nvidia_code/gangealing/models/__init__.py", line 3, in <module>
    from models.spatial_transformers.spatial_transformer import get_stn, ComposedSTN, SpatialTransformer
  File "/home/user/nvidia_code/gangealing/models/spatial_transformers/spatial_transformer.py", line 5, in <module>
    from models.stylegan2.networks import EqualLinear, ConvLayer, ResBlock
  File "/home/user/nvidia_code/gangealing/models/stylegan2/networks.py", line 6, in <module>
    from models.stylegan2.op import FusedLeakyReLU, fused_leaky_relu, upfirdn2d, conv2d_gradfix
  File "/home/user/nvidia_code/gangealing/models/stylegan2/op/__init__.py", line 1, in <module>
    from .fused_act import FusedLeakyReLU, fused_leaky_relu
  File "/home/user/nvidia_code/gangealing/models/stylegan2/op/fused_act.py", line 11, in <module>
    fused = load(
  File "/home/user/anaconda2/envs/gg/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 1144, in load
    return _jit_compile(
  File "/home/user/anaconda2/envs/gg/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 1357, in _jit_compile
    _write_ninja_file_and_build_library(
  File "/home/user/anaconda2/envs/gg/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 1456, in _write_ninja_file_and_build_library
    _write_ninja_file_to_build_library(
  File "/home/user/anaconda2/envs/gg/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 1857, in _write_ninja_file_to_build_library
    cuda_flags = common_cflags + COMMON_NVCC_FLAGS + _get_cuda_arch_flags()
  File "/home/user/anaconda2/envs/gg/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 1626, in _get_cuda_arch_flags
    arch_list[-1] += '+PTX'

THe command I run is : python applications/propagate_to_images.py --ckpt cat --real_data_path data/lsun_cats --real_size 512 --dset_indices 1922 2363 8558 7401 9750 7432 2105 53 1946

Any help would be appreciated.

Hi.
There is a RuntimeError: imageio.ffmpeg.download() has been deprecated. Use 'pip install imageio-ffmpeg' instead.' when running colab notebook:

Solution: add imageio==2.4.1 to dependencies or add !pip install imageio==2.4.1 to Setup cell.

Thanks for your great work

To run the vis_correspondence.py,
I setup this repo and preparing real data(LSUN cat) but the following error appear

Traceback (most recent call last):
  File "applications/propagate_to_images.py", line 154, in <module>
    make_visuals(args, t_ema, classifier)
  File "/home/ubuntu/anaconda3/envs/ugatit/lib/python3.8/site-packages/torch/autograd
/grad_mode.py", line 28, in decorate_context
    return func(*args, **kwargs)
  File "applications/propagate_to_images.py", line 49, in make_visuals
    sample_images_and_points(args, t, classifier, device='cuda')
  File "/home/ubuntu/anaconda3/envs/ugatit/lib/python3.8/site-packages/torch/autograd
/grad_mode.py", line 28, in decorate_context
    return func(*args, **kwargs)
  File "/data/hongiee/gangealing/applications/vis_correspondence.py", line 34, in sam
ple_images_and_points
 dset = MultiResolutionDataset(args.real_data_path, resolution=args.real_size)
  File "/data/hongiee/gangealing/datasets/dataset.py", line 28, in __init__
    self.length = int(txn.get('length'.encode('utf-8')).decode('utf-8'))
AttributeError: 'NoneType' object has no attribute 'decode'

how can I solve this problem?

Hello:

I've been using the model on my own custom dataset for a while. When I visualize the congealing process on test set and the propagated dense tracking, I noticed:

1. the congealing process synthesize features (i.e. create animal's head on its tail side), instead of rotate or flip the image to do the correct alignment
2. on the dense tracking, color scale will flip (i.e. flip by head to tail on animals), which I think corresponding to the last point

I read the paper about using flow smoothness and flip to avoid this issue and I understand this can occur a lot. How exactly are flip and flow smoothness helping avoid this issue? What parameter can I adjust to make my model more robust? Does the model improve on this issue after 1 million epochs ? For resources reason I haven't been able to run to 1 million epochs yet but I read the other issue post that you mention it usually takes that long for the model to improve.

I have try the default setting script for both 1, 2 head and 4 head. Then I also tried increase the inject and flow_size parameter. I also tried turn on the sample_from_full_resolution option, but haven't got any good progress from these trial yet.

Thanks in advance and it's really appriciated that you're consistently helping out : )

Hi:
I'm quite new from the STN model. Just like the title says, for every image that I align by using a trained STN, I want to get each of their predicted transformation (i.e., the predicted set of transformation matrices). Is there anyway I can extract that from the STN model ?

For some reason, I'm trying to use the predicted transformation of an image to apply on another different image. I saw in the forward function of STN there are return_flow, return_wrap function that can produce matrix, but not sure which should I take.

Thanks in advance : )

Hi again @wpeebles!

Looks like there is difference between the state_dict of the Generator and the pretrained weights for CelebA. Specifically:

RuntimeError: Error(s) in loading state_dict for Generator:
	Missing key(s) in state_dict: "convs.10.conv.weight", "convs.10.conv.blur.kernel", "convs.10.conv.modulation.weight", "convs.10.conv.modulation.bias", "convs.10.noise.weight", "convs.10.activate.bias", "convs.11.conv.weight", "convs.11.conv.modulation.weight", "convs.11.conv.modulation.bias", "convs.11.noise.weight", "convs.11.activate.bias", "to_rgbs.5.bias", "to_rgbs.5.upsample.kernel", "to_rgbs.5.conv.weight", "to_rgbs.5.conv.modulation.weight", "to_rgbs.5.conv.modulation.bias", "noises.noise_11", "noises.noise_12".

Looks like the generator has more StyledConv blocks and ToRGB blocks than the pretrained weights. Not sure which is supposed to be the correct configuration.

Command that I'm running:

bash scripts/training/celeba.sh 

Thanks for the help :)

Did anyone manage to run training on a single GPU? I keep getting torch.distributed.elastic.multiprocessing.errors.child failed error when using one GPU to train with this following training script:

torchrun --nproc_per_node=1 /path/train.py
--ckpt cat --load_G_only --padding_mode border --vis_every 500 --ckpt_every 10000
--iter 100000 --tv_weight 2500 --loss_fn lpips --exp-name lsun_cats_lpips --real_data_path /my/train_data/

Thanks in advance!

Edit: I tried to turn off the distributed mode but a math value error pops out in here

says Value error math domain error

Question:
Is the train_cluster_classifier.py independent of the train.py ?

Description
When I want to train a cluster model, I find the following code In train_cluster_classifier.py

# Load pre-trained STN, Generator and LL (required):
    print(f"Loading model from {args.ckpt}")
    ckpt = torch.load(args.ckpt, map_location=lambda storage, loc: storage)
    generator.load_state_dict(ckpt["g_ema"])
    t_ema.load_state_dict(ckpt["t_ema"])
    ll.load_state_dict(ckpt["ll"])

steps:

1. train a common model by the train.py to get a weight.pt
2. using the weight.pt, enhance the model by the train_cluster_classifier.py

Are the steps right?
In other words, must I use the train.py before using train_cluster_classifier.py ?

thanks！

Hello: Thank you so much for the awsome work and consistently helping with all the question

After I trained a model on my own dataset, I tried to run propagate_to_image.py with a mask I created my own to visualize the dense correspondence mask. but when it runs to building extension I got this error of RuntimeError: Error building extension '_splat' during visualization.

If I run without any mask (just congealing the input images) I get no errors.

I'm using cuda = 11.7 and pytorch = 1.11.0.

Is this likely be a cuda installation problem?

Thank you