Comments (15)
Thank you! We'll check your images!
from lama.
Hi! Sorry for the late reply.
Does this kind of artifacts appear always? Or maybe there are specific conditions where they are more noticeable than usual?
Basically, you can try setting training_model.image_to_discriminator=inpainted
- thus the discriminator and the losses will see a blended image gen_output * mask + input * (1 - mask)
instead of raw gen_output
.
from lama.
How big is your dataset? What is your training resolution and testing resolution?
from lama.
Hi, Sorry for the even later reply.
Does this kind of artifacts appear always? Or maybe there are specific conditions where they are more noticeable than usual?
It doesn't always appear: it is more present when the mask is over a solid color than on textures and meaningful content.
How big is your dataset? What is your training resolution and testing resolution?
The training set has 150000 images. The validation and test sets are 300 each. Resolution vary, with largest side being 1500px.
Basically, you can try setting training_model.image_to_discriminator=inpainted
I did one experiment so far, changing the config as you suggested, resuming the training on a trained model (for 100 epochs) and after 3, 10 and 30 epochs, the artifacts did not improve. I will further investigate and attempt a fresh training session with this blended image used for the discriminator loss.
Also, I was going to attempt adding 10000 generated images with random colors and gradients (and possibly similar jpeg encoding) to the next training session. Is this a naive approach ?
I also tested the proposed model big-lama and I encountered the same artifacts, using the proposed bin/predict.py script. I will attach 3 conclusive results. If you want to replicate this, let me know how you'd like me to attach the inputs.
from lama.
Hm... that's weird
In which resolution do you feed the images during training? Am I correct that you're applying Lama-Regular model to images of much higher resolution than during training?
I also tested the proposed model big-lama
Do you mean that a pre-trained (by us on Places) Big-Lama has same artifacts? Or you re-trained it on your data?
If you want to replicate this, let me know how you'd like me to attach the inputs.
Yes, a dozen of relevant images+masks would help a lot!
Thank you!
from lama.
Hi again,
In which resolution do you feed the images during training? Am I correct that you're applying Lama-Regular model to images of much higher resolution than during training?
You are right, I resize images to 512x512 during training and yes, I use Lama-Regular configuration. During prediction I don't resize, using original size ( which varies but it is around 1500).
Your assumption is probably correct, since I tested the same prediction on 512x512 images and the artifacts, while not entirely gone, are significantly reduced.
Do you mean that a pre-trained (by us on Places) Big-Lama has same artifacts? Or you re-trained it on your data?
Yes, I get similar artifacts using the model proposed by you, more exactly: "The best model (Places2, Places Challenge):", downloaded from:
curl -L $(yadisk-direct https://disk.yandex.ru/d/ouP6l8VJ0HpMZg) -o big-lama.zip
Below I attached 12 images and mask that should replicate the artifacts. If you prefer them in a different format, let me know.
I will also follow up with 1-2 more general questions, so if possible, don't close the Issue yet.
Thanks for taking the time.
from lama.
@windj007 I'm facing with the same problem after training 40 epoch on FFHQ dataset. However,the artifacts are much more obvious, any idea to alleviate the problem?
https://s2.loli.net/2022/03/08/ZiBMduyKPJporVF.jpg
from lama.
@ImmortalSdm is this image from training dataloader or from validation? If this is validation, then such artifacts may apper when the mask is non-binary, e.g. it has smooth transition between 0 (known areas) and 1 (missing areas).
from lama.
@eduardathome Sorry for the late reply again!
Have you tried training in 256?
Already after releasing the codebase we found that with Fourier-based generators training in 256 yields more robust performance than training in higher resolutions - that's most probably due to characteristics of the loss functions.
from lama.
@ImmortalSdm is this image from training dataloader or from validation? If this is validation, then such artifacts may apper when the mask is non-binary, e.g. it has smooth transition between 0 (known areas) and 1 (missing areas).
@ImmortalSdm is this image from training dataloader or from validation? If this is validation, then such artifacts may apper when the mask is non-binary, e.g. it has smooth transition between 0 (known areas) and 1 (missing areas).
yep, it's from validation. Thanks for your reply, i will check my mask image.
from lama.
As anyone solve this issue, I am also facing this
from lama.
Hi! I am facing some issues related with this. I am just fine tunning with LAMA. At first in order to do sannity check, I tried to just overfit to a single image of CelebHQ. When using the predict.py file, Everything works fine, but when training (overfitting) Saving the output of the network at different iterations I obtain this
In Here you can see the following. Top left - original image, Top right image with mask on it. bottom left output imagen of the network. Bottom right inpainted image
Input image size 512 with bs 1 (as I am overfitting to single image)
from lama.
When fine-tuning a pretrained model, please keep in mind:
- Overfitting to a single image with discriminator might break because it is too easy for discriminator to remember the real image exactly - so it wins and the training diverges
- When we tried resuming training from a checkpoint, we encountered instabilities when batch size or number of gpus was changed after restart compared to those before. I do not know exactly why, but most probably due to batchnorm or adam stats.
from lama.
@ImmortalSdm Hello! Did you manage to train lama on ffhq in the end? I am having issues and I would so greatly appreciate a trianed checkpoint if you managed.
from lama.
Hi again,
In which resolution do you feed the images during training? Am I correct that you're applying Lama-Regular model to images of much higher resolution than during training?
You are right, I resize images to 512x512 during training and yes, I use Lama-Regular configuration. During prediction I don't resize, using original size ( which varies but it is around 1500).
Your assumption is probably correct, since I tested the same prediction on 512x512 images and the artifacts, while not entirely gone, are significantly reduced.
Do you mean that a pre-trained (by us on Places) Big-Lama has same artifacts? Or you re-trained it on your data?
Yes, I get similar artifacts using the model proposed by you, more exactly: "The best model (Places2, Places Challenge):", downloaded from:
curl -L $(yadisk-direct https://disk.yandex.ru/d/ouP6l8VJ0HpMZg) -o big-lama.zip
Below I attached 12 images and mask that should replicate the artifacts. If you prefer them in a different format, let me know.
I will also follow up with 1-2 more general questions, so if possible, don't close the Issue yet.
Thanks for taking the time.
@eduardathome Have you solved the problem? I have the same issues as you. I would so greatly appreciate if you share your solution.
from lama.
Related Issues (20)
- Hi, I have made a iOS App with your great model!
- Prediction failed due to Missing key visualizer
- Can I fine-tune the model? HOT 2
- why is tensorflow necessary?
- ImportError: cannot import name 'DualIAATransform' from 'albumentations' HOT 1
- About the training command 2 HOT 1
- Created single-file version of LaMa
- Question about generating validation and eval data
- Can I separate the Feature Refinement to Improve the High-Resolution Image Inpainting technique
- A simple ckpt to pt model convertor
- Repeated Refinement?
- Error finetuning the big-lama-with-discr model HOT 7
- Data set training problem HOT 1
- After executing the training command, it has been stuck at this point without any progress in the training. HOT 1
- Inpaint a NEW thing? HOT 3
- Refinement with Multiple Images
- How to draw a loss function curve
- Dataset is empty if configuring img_suffix: .jpg in default.yaml
- ONNX Model done HOT 10
- Output Error: No inpainted in the output_dir HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from lama.