dave-epstein / blobgan Goto Github PK
View Code? Open in Web Editor NEWOfficial PyTorch implementation of BlobGAN: Spatially Disentangled Scene Representations
License: BSD 2-Clause "Simplified" License
Official PyTorch implementation of BlobGAN: Spatially Disentangled Scene Representations
License: BSD 2-Clause "Simplified" License
Thank you for providing the code. Your work is very interesting.
Do you intend to provide the weights of the encoder for image inversion? on bedroom for example?
Thank you
Thanks for your great work and well-organized code! Could you please also share the restyling and cloning code? I tried restyling, but I'm not sure what's the thin border of the blob; I also tried cloning by simply copying all the parameters of one blob, but I can't get the same object on the image.
Line 185 in 886a44b
Hi. I was trying to replicate the experiment with my custom dataset of bedrooms. I've modified my local.yaml
and set the basepath
to the image folder, where it contains train
, test
, and validate
. I'm sure that they all contain images. But when I execute this:
python src/run.py +experiment=[blobgan,local,jitter] wandb.name='10-blob BlobGAN on bedrooms'
It gives this error:
AssertionError: Cannot compute FID without name of statistics file to use.
After scrolling up, I found these errors:
Warning: no images found in /home/PythonProjects/blobgan/resizedout/train. Using empty dataset for split train. Perhaps you set `dataset.path` incorrectly?
Warning: no images found in /home/PythonProjects/blobgan/resizedout/validate. Using empty dataset for split validate. Perhaps you set `dataset.path` incorrectly?
Warning: no images found in /home/PythonProjects/blobgan/resizedout/test. Using empty dataset for split test. Perhaps you set `dataset.path` incorrectly?
This is the entire error log:
resume:
id: null
step: null
epoch: null
last: true
best: false
clobber_hparams: false
project: Blobgan investigation
log_dir: ./logs
model_only: false
logger: wandb
wandb:
save_code: true
offline: false
log_dir: ./logs
id: null
name: 10-blob BlobGAN on bedrooms
group: XXX Group
project: Blobgan investigation
entity: yimingsu
trainer:
accelerator: ddp
benchmark: false
deterministic: false
gpus: 1
precision: 32
plugins: null
max_steps: 10000000
profiler: simple
num_sanity_val_steps: 0
log_every_n_steps: 200
limit_val_batches: 0
dataset:
dataloader:
prefetch_factor: 2
pin_memory: true
drop_last: true
persistent_workers: true
num_workers: 12
batch_size: 24
name: ImageFolderDataModule
resolution: 256
category: bedroom
path: /home/yimingsu/PythonProjects/blobgan/resizedout/
mode: fit
seed: 0
checkpoint:
every_n_train_steps: 1500
save_top_k: -1
mode: max
monitor: step
model:
name: BlobGAN
lr: 0.002
dim: 512
noise_dim: 512
resolution: 256
lambda:
D_real: 1
D_fake: 1
D_R1: 50
G: 1
G_path: 2
G_feature_mean: 10
G_feature_variance: 10
discriminator:
name: StyleGANDiscriminator
size: 256
generator:
name: models.networks.layoutstylegan.LayoutStyleGANGenerator
style_dim: 512
n_mlp: 8
size_in: 16
c_model: 96
spatial_style: true
size: 256
layout_net:
name: models.networks.layoutnet.LayoutGenerator
n_features_max: 10
feature_dim: 768
style_dim: 512
noise_dim: 512
norm_features: true
mlp_lr_mul: 0.01
mlp_hidden_dim: 1024
spatial_style: true
D_reg_every: 16
G_reg_every: -1
λ:
D_real: 1
D_fake: 1
D_R1: 50
G: 1
G_path: 2
G_feature_mean: 10
G_feature_variance: 10
log_images_every_n_steps: 1000
n_features_min: 10
n_features_max: 10
n_features: 10
spatial_style: true
feature_jitter_xy: 0.04
feature_jitter_shift: 0.5
feature_jitter_angle: 0.1
Global seed set to 0
wandb: Currently logged in as: yimingsu (use `wandb login --relogin` to force relogin)
wandb: wandb version 0.13.2 is available! To upgrade, please run:
wandb: $ pip install wandb --upgrade
wandb: Tracking run with wandb version 0.12.11
wandb: Run data is saved locally in /home/yimingsu/PythonProjects/blobgan/logs/wandb/run-20220902_114754-xjimkxit
wandb: Run `wandb offline` to turn off syncing.
wandb: Syncing run 10-blob BlobGAN on bedrooms
wandb: ⭐️ View project at https://wandb.ai/yimingsu/Blobgan%20investigation
wandb: 🚀 View run at https://wandb.ai/yimingsu/Blobgan%20investigation/runs/xjimkxit
[2022-09-02 11:47:58,898][torch.distributed.nn.jit.instantiator][INFO] - Created a temporary directory at /tmp/tmpr5cdgp32
[2022-09-02 11:47:58,898][torch.distributed.nn.jit.instantiator][INFO] - Writing /tmp/tmpr5cdgp32/_remote_module_non_sriptable.py
Froze 65 parameters - ['conv1.conv', 'conv1.noise', 'conv1.activate', 'to_rgb1', 'to_rgb1.conv', 'convs.0', 'convs.1', 'convs.2', 'convs.3', 'convs.4', 'convs.5', 'convs.6', 'convs.7', 'to_rgbs.0', 'to_rgbs.1', 'to_rgbs.2', 'to_rgbs.3'] - for model of type LayoutStyleGANGenerator
Froze 16 parameters - ['mlp.1', 'mlp.2', 'mlp.3', 'mlp.4', 'mlp.5', 'mlp.6', 'mlp.7', 'mlp.8'] - for model of type LayoutGenerator
[2022-09-02 11:47:59,637][py.warnings][WARNING] - /home/yimingsu/anaconda3/envs/blobgan/lib/python3.9/site-packages/pytorch_lightning/trainer/connectors/accelerator_connector.py:286: LightningDeprecationWarning: Passing `Trainer(accelerator='ddp')` has been deprecated in v1.5 and will be removed in v1.7. Use `Trainer(strategy='ddp')` instead.
rank_zero_deprecation(
[2022-09-02 11:47:59,637][py.warnings][WARNING] - /home/yimingsu/anaconda3/envs/blobgan/lib/python3.9/site-packages/pytorch_lightning/trainer/connectors/callback_connector.py:147: LightningDeprecationWarning: Setting `Trainer(checkpoint_callback=True)` is deprecated in v1.5 and will be removed in v1.7. Please consider using `Trainer(enable_checkpointing=True)`.
rank_zero_deprecation(
GPU available: True, used: True
TPU available: False, using: 0 TPU cores
IPU available: False, using: 0 IPUs
Global seed set to 0
initializing distributed: GLOBAL_RANK: 0, MEMBER: 1/1
[2022-09-02 11:47:59,641][torch.distributed.distributed_c10d][INFO] - Added key: store_based_barrier_key:1 to store for rank: 0
[2022-09-02 11:47:59,641][torch.distributed.distributed_c10d][INFO] - Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
----------------------------------------------------------------------------------------------------
distributed_backend=nccl
All distributed processes registered. Starting with 1 processes
----------------------------------------------------------------------------------------------------
Warning: no images found in /home/yimingsu/PythonProjects/blobgan/resizedout/train. Using empty dataset for split train. Perhaps you set `dataset.path` incorrectly?
Warning: no images found in /home/yimingsu/PythonProjects/blobgan/resizedout/validate. Using empty dataset for split validate. Perhaps you set `dataset.path` incorrectly?
Warning: no images found in /home/yimingsu/PythonProjects/blobgan/resizedout/test. Using empty dataset for split test. Perhaps you set `dataset.path` incorrectly?
LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]
Optimizing 57.19M params for G and 28.86M params for D
| Name | Type | Params
-----------------------------------------------------------
0 | discriminator | StyleGANDiscriminator | 28.9 M
1 | generator_ema | LayoutStyleGANGenerator | 35.9 M
2 | generator | LayoutStyleGANGenerator | 35.9 M
3 | layout_net_ema | LayoutGenerator | 21.3 M
4 | layout_net | LayoutGenerator | 21.3 M
-----------------------------------------------------------
86.1 M Trainable params
57.2 M Non-trainable params
143 M Total params
573.008 Total estimated model params size (MB)
[2022-09-02 11:48:01,076][py.warnings][WARNING] - /home/yimingsu/anaconda3/envs/blobgan/lib/python3.9/site-packages/torch/utils/data/dataloader.py:487: UserWarning: This DataLoader will create 12 worker processes in total. Our suggested max number of worker in current system is 8, which is smaller than what this DataLoader is going to create. Please be aware that excessive worker creation might get DataLoader running slow or even freeze, lower the worker number to avoid potential slowness/freeze if necessary.
warnings.warn(_create_warning_msg(
Training: 0it [00:00, ?it/s]Error executing job with overrides: ['+experiment=[blobgan,local,jitter]', 'wandb.name=10-blob BlobGAN on bedrooms', 'dataset=imagefolder', '++dataset.path=/home/yimingsu/PythonProjects/blobgan/resizedout/']
Traceback (most recent call last):
File "/home/yimingsu/PythonProjects/blobgan/src/run.py", line 81, in run
trainer.fit(model, datamodule=datamodule)
File "/home/yimingsu/anaconda3/envs/blobgan/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 740, in fit
self._call_and_handle_interrupt(
File "/home/yimingsu/anaconda3/envs/blobgan/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 685, in _call_and_handle_interrupt
return trainer_fn(*args, **kwargs)
File "/home/yimingsu/anaconda3/envs/blobgan/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 777, in _fit_impl
self._run(model, ckpt_path=ckpt_path)
File "/home/yimingsu/anaconda3/envs/blobgan/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 1199, in _run
self._dispatch()
File "/home/yimingsu/anaconda3/envs/blobgan/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 1279, in _dispatch
self.training_type_plugin.start_training(self)
File "/home/yimingsu/anaconda3/envs/blobgan/lib/python3.9/site-packages/pytorch_lightning/plugins/training_type/training_type_plugin.py", line 202, in start_training
self._results = trainer.run_stage()
File "/home/yimingsu/anaconda3/envs/blobgan/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 1289, in run_stage
return self._run_train()
File "/home/yimingsu/anaconda3/envs/blobgan/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 1319, in _run_train
self.fit_loop.run()
File "/home/yimingsu/anaconda3/envs/blobgan/lib/python3.9/site-packages/pytorch_lightning/loops/base.py", line 140, in run
self.on_run_start(*args, **kwargs)
File "/home/yimingsu/anaconda3/envs/blobgan/lib/python3.9/site-packages/pytorch_lightning/loops/fit_loop.py", line 200, in on_run_start
self.trainer.call_hook("on_train_start")
File "/home/yimingsu/anaconda3/envs/blobgan/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 1501, in call_hook
output = model_fx(*args, **kwargs)
File "/home/yimingsu/PythonProjects/blobgan/src/models/blobgan.py", line 131, in on_train_start
assert not ((self.log_fid_every_n_steps > -1 or self.log_fid_every_epoch) and (not self.fid_stats_name)), \
AssertionError: Cannot compute FID without name of statistics file to use.
Set the environment variable HYDRA_FULL_ERROR=1 for a complete stack trace.
Any help would be greatly appreciated!
Best,
Yiming
Hi,
I have a question regarding image inversion. To train the encoder, did you use the jittering of the spatial parameters? In the 'invertblobgan.py' code no_jitter is not used and the invertblobgan.yaml doesn't set the jittering parameters to 0. Is this intended?
Hi,
I cannot reproduce with the same layout
layout, img = model.gen(z, ret_layout=True)
layout, img = model.gen(layout=layout, ret_layout=True)
Line 303 in c527f1c
Line 323 in c527f1c
I am wondering the basis computation is wrong here, which you will get a tensor with shape BxCx2x0
:
Line 327 in c527f1c
Hello,
I have some questions regarding paragraph 3.1 of the paper:
Best,
Mehdi
Hi.
While reading the paper, I'm a little bit confused of the training time for the BlobGAN.
Specifically, in the Appendix E.1 in the paper, it is written as "We train all models for 1.5 million gradient steps with batch size 24 per-GPU across 8 NVIDIA A100 GPUs. This takes around 4 weeks."
Now I'm wondering if it takes 4 weeks with 8 NVIDIA A100 GPUs for training a single model or for all models.
Could you please clarify this more?
Thanks!
Hi,
I'm getting this error when trying to train my own model on multiple GPUs.
This is my command:
python src/run.py +experiment=[blobgan,local,jitter] wandb.name='10-blob BlobGAN on bedrooms
This is my local.yaml:
dataset:
path: /home/yimingsu/blobgan/resizedout # Change to your path
resolution: 128
#dataloader:
#batch_size: 24
trainer:
gpus: 4 # Change to your number of GPUs
wandb: # Fill in your settings
group: Group
project: Blobgan investigation
entity: yimingsu
model:
#layout_net:
#feature_dim: 256
#generator:
#override_c_in: 256
#dim: 256
fid_stats_name: mitbedroom # this is my custom dataset.
This is the error log:
Error executing job with overrides: ['+experiment=[blobgan,local,jitter]', 'wandb.name=10-blob BlobGAN on bedrooms']
Traceback (most recent call last):
File "/home/yimingsu/blobgan/src/run.py", line 81, in run
trainer.fit(model, datamodule=datamodule)
File "/home/yimingsu/blobganenv/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 740, in fit
self._call_and_handle_interrupt(
File "/home/yimingsu/blobganenv/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 685, in _call_and_handle_interrupt
return trainer_fn(*args, **kwargs)
File "/home/yimingsu/blobganenv/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 777, in _fit_impl
self._run(model, ckpt_path=ckpt_path)
File "/home/yimingsu/blobganenv/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 1138, in _run
self._call_setup_hook() # allow user to setup lightning_module in accelerator environment
File "/home/yimingsu/blobganenv/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 1435, in _call_setup_hook
self.training_type_plugin.barrier("pre_setup")
File "/home/yimingsu/blobganenv/lib/python3.9/site-packages/pytorch_lightning/plugins/training_type/ddp.py", line 403, in barrier
torch.distributed.barrier(device_ids=self.determine_ddp_device_ids())
File "/home/yimingsu/blobganenv/lib/python3.9/site-packages/torch/distributed/distributed_c10d.py", line 2784, in barrier
work = default_pg.barrier(opts=opts)
RuntimeError: CUDA error: out of memory
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Set the environment variable HYDRA_FULL_ERROR=1 for a complete stack trace.
I'm training on 4 RTX 2080 Ti so I don't think the memory is the actual issue.
Any help would be appreciated!
Hi,
Thanks for the repo. It is pretty interesting to play around.
I am wondering how do I edit the blobs to make things vanish? Presumably, I will be editing the outputs from the inversion models, right? So, how should I do that?
Best,
Jian
In appendix E.1 Hyperparameters of your paper you mentioned that you used different model configurations depending on the dataset (number of objects per image).
I want to train a model with k=40, d_in= 256 and d_style=256. I changed output shape of the layout network according to these values and 'c_model' of the generator to 32 to match to spatial_style number of channels.
This is my config file:
model:
name: BlobGAN
lr: 0.002
dim: 256
noise_dim: 512
resolution: 256
lambda:
D_real: 1
D_fake: 1
D_R1: 50
G: 1
G_path: 2
G_feature_mean: 10
G_feature_variance: 10
discriminator:
name: StyleGANDiscriminator
size: ${model.resolution}
generator:
name: models.networks.layoutstylegan.LayoutStyleGANGenerator
style_dim: ${model.dim}
n_mlp: 8
size_in: 16
c_model: 32
spatial_style: ${model.spatial_style}
size: ${model.resolution}
layout_net:
name: models.networks.layoutnet.LayoutGenerator
n_features_max: ${model.n_features_max}
feature_dim: 256
style_dim: ${model.dim}
noise_dim: ${model.noise_dim}
norm_features: true
mlp_lr_mul: 0.01
mlp_hidden_dim: 1024
spatial_style: ${model.spatial_style}
n_features_min: ${model.n_features}
n_features_max: ${model.n_features}
n_features: 40
spatial_style: true
Hower with this configuration I get very few parameters in the generator:
| Name | Type | Params
0 | discriminator | StyleGANDiscriminator | 28.9 M
1 | generator_ema | LayoutStyleGANGenerator | 4.2 M
2 | generator | LayoutStyleGANGenerator | 4.2 M
3 | layout_net_ema | LayoutGenerator | 28.6 M
4 | layout_net | LayoutGenerator | 28.6 M
61.7 M Trainable params
32.9 M Non-trainable params
94.6 M Total params
378.321 Total estimated model params size (MB)
Hi Dr. Epstein,
Thank you for this great work and published codes.
I tried to manipulate blobs using your BlobGAN Playground Colab (Last cell: Interactively edit scenes). I also ran the code on the server.
They both gave the same error message
RuntimeError Traceback (most recent call last)
in <cell line: 37>()
35 layout, orig_img = model.gen(z=z, truncate=truncate, layout=layout, **render_kwargs)
36 orig_blobs = for_canvas(layout['feature_img'].mul(255))
---> 37 labeled_blobs, labeled_blobs_img = draw_labels(orig_blobs, layout, size_threshold, COLORS)
38
39 blobs = DraggableBlobMap(locals())
/content/blobgan/./src/utils/colab.py in draw_labels(img, layout, T, colors, layout_i)
88 # draw = ImageDraw.Draw(img)
89 # mask = layout['sizes'][layout_i, 1:] > T
---> 90 # idmap = torch.arange(len(mask))[mask]
91 # blob = {k: layout[k][layout_i][mask].mul(255).tolist() for k in ('xs', 'ys')}
92 # for i, (x, y) in enumerate(zip(blob['xs'], blob['ys'])):
RuntimeError: indices should be either on cpu or on the same device as the indexed tensor (cpu)
I searched online and no solutions work for me. Could you please tell me how to solve this for manipulating blobs and generating the corresponding images?
I appreciate any help from you.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.