mchong6 / soat Goto Github PK

View Code? Open in Web Editor NEW

374.0 11.0 57.0 84.3 MB

Official PyTorch repo for StyleGAN of All Trades: Image Manipulation with Only Pretrained StyleGAN.

License: MIT License

Jupyter Notebook 97.42% Python 2.58%

soat's Introduction

StyleGAN of All Trades: Image Manipulation with Only Pretrained StyleGAN

This is the PyTorch implementation of StyleGAN of All Trades: Image Manipulation with Only Pretrained StyleGAN.

Web Demo Integrated to Huggingface Spaces with Gradio. See demo for Panorama Generation for Landscapes:

Abstract:
Recently, StyleGAN has enabled various image manipulation and editing tasks thanks to the high-quality generation and the disentangled latent space. However, additional architectures or task-specific training paradigms are usually required for different tasks. In this work, we take a deeper look at the spatial properties of StyleGAN. We show that with a pretrained StyleGAN along with some operations, without any additional architecture, we can perform comparably to the state-of-the-art methods on various tasks, including image blending, panorama generation, generation from a single image, controllable and local multimodal image to image translation, and attributes transfer.

How to use

Everything to get started is in the colab notebook.

Toonification

For toonification, you can train a new model yourself by running

python train.py

For disney toonification, we use the disney dataset here. Feel free to experiment with different datasets.

GAN inversion

To perform GAN inversion with gaussian regularization in W+ space,

python projector.py xxx.jpg

the code will be saved in ./inversion_codes/xxx.pt which you can load by

source = load_source(['xxx'], generator, device)
source_im, _ = generator(source)

Citation

If you use this code or ideas from our paper, please cite our paper:

@article{chong2021stylegan,
  title={StyleGAN of All Trades: Image Manipulation with Only Pretrained StyleGAN},
  author={Chong, Min Jin and Lee, Hsin-Ying and Forsyth, David},
  journal={arXiv preprint arXiv:2111.01619},
  year={2021}
}

Acknowledgments

This code borrows from StyleGAN2 by rosalinity

soat's People

Contributors

Stargazers

Watchers

Forkers

pixel-lt ricklentz saulocatharino josebrunods moileehyeji yotofu peterzhousz cv-ip crysist-sui jankin-github asdlei99 as85207 ray-wzm gschmagee rhinojosa slurdge ak391 yangkk2019 bmd080 gouxiayibu liuguoyou kkodoo tonyfan08 napohou jinwook-shim kang-hana xiao-hua-sheng gdbrianlu arpit-dhamija xiaojize celsopitta 0xpussycat knut0815 alixing mornydew reed6868 kairess stevenhailin leixiaoning poem4love hunsii superwhyun hanhduyenjn ernestico98 haizhu12 liyunfei0411

soat's Issues

What operations does the display_image method perform?

Thanks, I find out this method successfully! ——From the second edit

the detail about how to transform the latent code into image？

StyleGAN inversion

Hello! Thanks for the work done, the results look great. I was particularly impressed by your image inversion, but I am not quite sure how it works. Do you plan to publish the relevant code?

How to use pkl files of official stylegan?

How to use pkl files of official stylegan2 or stylegan3?

Stuck on the Colab execution.

When I start running the colab file, it stop responding in

freeman@freeman-T430s:~/SOAT/models/research$ python -m pip install -q .
  DEPRECATION: A future pip version will change local packages to be built in-place without first copying to a temporary directory. We recommend you use --use-feature=in-tree-build to test your packages with this new behavior before it becomes the default.
   pip 21.3 will remove support for this functionality. You can find discussion regarding this at https://github.com/pypa/pip/issues/7555.

^CERROR: Operation cancelled by user

It occurs on my local laptop too. What should I do ?

How to use custom pictures?

Hello, can you use custom pictures for reasoning?

How to use my picture for toonify?

Hello, is it possible to use your own images, not generated, for toonification?

about face and disney model

Thank you for your nice work,!
Can you describe the source of these two models:disney.pt, face.pt?

Transfer multiple features from image to image using bbox

I'd like to annotate and transfer multiple bboxes from the target image to the source image. I noticed in running infinity.ipynb that while I can use Colab to annotate multiple bboxes on the target, only the first of them gets transferred from the source.

This behavior seems consistent with the blend_bbox code in model.py where only the first bbox (coord[0]) is being considered and the rest don't seem to be processed.

Two questions - a) am I missing something here and b) if the above is accurate, is it useful if I add an outer loop to enable blend_bbox to iterate thru the bbox_list

cpu inference colab

is it possible to due inference on cpu?

CUDA Out Of Memory in Distributed Training

I used to successfully train the StyleGAN2-ADA and StyleGAN3 on my device. However the distributed training for SOAT failed due to out of the cuda memory. I modify the code a little bit which don't involving any training codes, then I use the Slurm to submit my training job to the server and check the model has been successfully distributed to different GPUs. Before the first epoch completes, the job aborts.
The information below is my training environment:
    CPU: Intel Xeon 6348
    GPU: NVIDIA A100 40G PCIe*8
    Script:  python -m torch.distributed.launch --nproc_per_node=8 train.py --dataset=[My Dataset(Grayscale in 1024x1024, and I convert it into RGB when loading dataset)] --batch=X --size=1024 --iter=40000
BTW, I set the batch size as 64, 32, 16. All of them abort. When I using a single GPU to train the SOAT with batch size 8, it succeeds.
Looking for your reply and see if there's any possible solution.

use e4e latent

is possible load a e4e latent?