jingyuanli001 / rfr-inpainting Goto Github PK

Hello, I have some problem about training your model.
What should I put in the folder of data_root and mask_root?

Hi, thanks for your shared work.
I try to train a new model with some additional masks in Celeba-HQ, and I found there is no STOP ITERS NUMBER for training/fine-tuning process. Could you plz share when I can finish the training/fine-tuning?

Hi, I get some awesome results on Celeba dataset using your released model. But I dont know if I sample the images from training set. So can you release your testing set list in Celeba?

特别想问一下您CelebA的训练集18w张图片和测试集如何划分呢？
使用CelebA训练的时候文件夹选择 img_align_celeba、img_align_celeba_png.7z 还是 img_celeba.7z ?

It appears the code only support RGB images. Can you add support for grayscale too?

when i train/test model,arise some problem about load pretrained model .The code in model.py as follow
def initialize_model(self, path=None, train=True):
self.G = RFRNet()
if train:
self.lossNet = VGG16FeatureExtractor()
self.optm_G = optim.Adam(self.G.parameters(), lr = 2e-4)
try:
start_iter = load_ckpt(path, [('generator', self.G)], [('optimizer_G', self.optm_G)])
print("here")
if train:
self.optm_G = optim.Adam(self.G.parameters(), lr = 2e-4)
print('Model Initialized, iter: ', start_iter)
self.iter = start_iter
except:
print('No trained model, from start')

,and in utils.io---->load_ckpt(),the code as follow

if optimizers is not None:
for prefix, optimizer in optimizers:
optimizer.load_state_dict(ckpt_dict[prefix])

may self.optm_G = optim.Adam(self.G.parameters(), lr = 2e-4) should be writen in try???otherwise self.optm_G will be None ,and pretrained model cannot be load .

Thanks for sharing this awesome work.
Can you please provide pre-trained model weights so that we can finetune your model.

i have trained RFRNet on my own dataset, i got great trainng result ,but the test result is terrible, the difference between trainset and testset is different mask , could u help me?

when --test mode, out of memory at the second image

Hi jingyuan, I would like to know the mask ratio of the pretrain model for CelebA. Have you used 40%-50% to train the model? And are the results under different mask ratio reported in your paper tested using a single pretrain model, or have you trained several models for different mask ratios. Thx!

Could you provide the code of Supplementary Material?Thank you!

hello @blmoistawinde @jingyuanli001
training command:

python run.py \
--data_root ./train_images \
--mask_root ./train_masks \
--model_save_path ./output_weights/test_iter_600500.pth \
--result_save_path ./training_results/ \
--model_path ./pre_trained_weights/checkpoint_celeba.pth \
--target_size 224 \
--mask_mode 0 \
--batch_size 5 \
--gpu_id 0 \
--num_iters 600050 

getting this error:

  File "run.py", line 38, in <module>
    run()
  File "run.py", line 35, in run
    model.train(dataloader, args.model_save_path, args.finetune, args.num_iters)
  File "/media/ai/e1f5ec44-04e5-413d-b816-57a5173d06c528/ai/Anglo_American/ACL/image_inpainting_GAN/research/RFR-Inpainting/model.py", line 58, in train
    for items in train_loader:
  File "/home/ai/anaconda3/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 345, in __next__
    data = self._next_data()
  File "/home/ai/anaconda3/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 385, in _next_data
    data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
  File "/home/ai/anaconda3/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py", line 47, in fetch
    return self.collate_fn(data)
  File "/home/ai/anaconda3/lib/python3.8/site-packages/torch/utils/data/_utils/collate.py", line 79, in default_collate
    return [default_collate(samples) for samples in transposed]
  File "/home/ai/anaconda3/lib/python3.8/site-packages/torch/utils/data/_utils/collate.py", line 79, in <listcomp>
    return [default_collate(samples) for samples in transposed]
  File "/home/ai/anaconda3/lib/python3.8/site-packages/torch/utils/data/_utils/collate.py", line 55, in default_collate
    return torch.stack(batch, 0, out=out)
RuntimeError: stack expects each tensor to be equal size, but got [3, 224, 224] at entry 0 and [4, 224, 224] at entry 3

Hi, thanks for your work.
I have some problems on CelebA.

Traceback (most recent call last):
  File "F:/PyTorch-CV/RFR-Inpainting-master/test.py", line 39, in <module>
    run()
  File "F:/PyTorch-CV/RFR-Inpainting-master/test.py", line 29, in run
    model.initialize_model(args.model_path, False)
  File "F:\PyTorch-CV\RFR-Inpainting-master\model_test.py", line 31, in initialize_model
    start_iter = load_ckpt(path, [('generator', self.G)], [('optimizer_G', self.optm_G)])
  File "F:\PyTorch-CV\RFR-Inpainting-master\utils\io.py", line 25, in load_ckpt
    ckpt_dict = torch.load(ckpt_name)
  File "E:\anaconda\anaconda\envs\pytorch_gpu\lib\site-packages\torch\serialization.py", line 529, in load
    return _legacy_load(opened_file, map_location, pickle_module, **pickle_load_args)
  File "E:\anaconda\anaconda\envs\pytorch_gpu\lib\site-packages\torch\serialization.py", line 704, in _legacy_load
    deserialized_storage_keys = pickle_module.load(f, **pickle_load_args)
_pickle.UnpicklingError: pickle data was truncated

dear @jingyuanli001 , I really appreciate your great job. The result in your paper is amazing. Then I test my own image with your released pretrained model 'RFR_Net_pretrained/checkpoint_celeba.pth'. But my result is bad. Can you help me?
the input are:

if I use my own mask, the result:

or:

if use the random mask(set mask_mode=1), the result:

dose it depends on the masks inputed?

Trying to train the model using the fp16 setting,

but the model outputs -inf's during the forward pass.

If I try to train the model, training starts and then the script immediately exists with no error message.

> python run.py --data_root data_train/images --mask_root data_train/masks --model_path checkpoints/checkpoint_celeba.pth --target_size 256 --mask_mode 1 --batch_size 6 --gpu 0
Model Initialized, iter:  600000
Model moved to cuda
Starting training from iteration:600000

There is a new file checkpoint/g_final.pth, but apart from that nothing seems to happen and the script exits.

Hello! I'm interested in your paper. Thanks for your pre_trained model.
When loading the pre_trained model for testing results, I found that the model couldn't load. My verification code is as follows：
import torch
a = torch.load('/home/sq/Codes/RFR-Inpainting-master/pre_model/checkpoint_celeba.pth')

The results of terminal operation as follows:

Traceback (most recent call last):
File "111.py", line 3, in
a = torch.load('/home/sq/Codes/RFR-Inpainting-master/pre_model/checkpoint_celeba.pth')
File "/home/sq/anaconda3/envs/outpainting/lib/python3.6/site-packages/torch/serialization.py", line 387, in load
return _load(f, map_location, pickle_module, **pickle_load_args)
File "/home/sq/anaconda3/envs/outpainting/lib/python3.6/site-packages/torch/serialization.py", line 576, in _load
deserialized_storage_keys = pickle_module.load(f, **pickle_load_args)
_pickle.UnpicklingError: pickle data was truncated

Environment：
python=3.6 pytorch=1.1.0

Do you have this problem? Or there are some errors about this pre_trained model.
Thanks.

when i test with python run.py --data_root examples/psv/images --mask_root examples/psv/masks --model_path pretrainedModel/checkpoint_paris.pth --test ,

It only shows one sentence Model moved to cuda ,
Other than that, there are no hints or results

The problem is here:

The final line (save_ckpt) should not be inside the if.

First of all, thank you for your work!

I tried the CelebA pretrained weight on some pictures that were cropped from pictures in the CelebA dataset. These cropped pictures are mostly of faces. However, I found that though the model worked well on pictures originally from the CelebA dataset, it did not give out a good result when cropped pictures were provided. Masks are randomly added to my tests.

For example, here are 2 pictures from the original CelebA dataset.

here are the masked of these 2 pictures

The model's results on these 2 pictures are indeed good.

Here are my cropped version of those 2 pictures. I chose the face parts and cropped them out.

here are the masked of these 2 cropped pictures

However,the results this time are not very good, even they were masked to a similar extent.

Considering the result, the model might be overfitting? Looking forward to your reply on this test.

Thanks for your great work! I have trained your model with my own dataset. However, I found sometimes there will be a gray region in the output, which is the center region of masked area. The attached images is one example.

How can I improve this?

EageConnect is ordinary convolution, not partial convolution，Please tell me how to replace it, thank you very much

Hi... I had tried to run your code, but i had this kind of issue!
When I run the code, I have no problem and it's created the checkpoint path with this argument (python run.py), but when I put the argument to test (python run.py --test), I have no problem too, but its not created the result path with the regenerated images.
Do you now whats wrong with my attempt?

Dear @jingyuanli001 , hi, I've been training your model with the places2 large dataset for quite some time, over few million iterations, and I've been seeing the same weird triangle artifact appearing in many images.

Some modifications I've done are:

• Increasing image training size to: 480
• Increasing hole size by multiplying max_width and max_length both by 1.5 in the random hole generation function.
• Increasing the RFR iterations or recurrence from 6 to 13 to account for the larger image and holes.

I think these artifacts appear when the holes are bigger. Have you encountered this problem? Please help, thank you.

Thanks for your work,fristly!!
I have some works about image inpainting , and appreciate your help if you can provide pretrained model for test?
emial:[email protected]

Hi，Thank you for your great project again！
How do you divide the masks training dataset from pconv? Whether to use all masks training data sets as training data sets ?

First of all, thanks for your wonderful work!
I think the picture should keep its original size during the test process. But in the process, I found a bug in the 17th line of “modules/Attention.py”. I suppose it should be "bz, nc, h, w = foreground.size()"

when I test with 256x256, it takes 4s with cpu, and if I test with 512x512, it takes 25s in cpu.

Hello, whenver I try to test a pretrained model, I get the following error:

File "D:\AI\RFR-Inpainting\modules\partialconv2d.py", line 55, in forward
    self.update_mask = F.conv2d(mask, self.weight_maskUpdater, bias=None, stride=self.stride, padding=self.padding, dilation=self.dilation, groups=1)
RuntimeError: Given groups=1, weight of size [64, 3, 7, 7], expected input[1, 9, 256, 256] to have 3 channels, but got 9 channels instead

Hi,

Thank you very much for sharing your awesome work. I am wondering if you plan to share the Places2 weight?

Dear @jingyuanli001 , I tried to train a model using the places2 dataset. After I have trained for approximately 750,000 iterations (batch size 9 using 3 GPUs; size 384), then I extracted the checkpoint and test it on the coco dataset. I'm getting unsatisfactory results. Why do you think there are obvious "lime water" effects around the masked regions?

The testing masks and the training masks are the same.

link

Two attributes 'stride' and 'prop_kernels' doesn't seem to be used. Can I remove them?

link 2

Line 150 seems to be redundant with the existence of line 151, please correct me if I'm wrong.

I am not sure the results are reliable. Will u release Celeba-hq models?

hi there I trained bird dataset for 800000 iters，yet the results remain blur. Does this have somthing todo with my mask setting?
I opened mask with mode='RGB' and deleted the code 'masks = torch.cat([masks]*3, dim = 1)'

Hi,

First of all, awesome work. Thank you.

Can we train the model on rectangular images? For example, 128x256?

Thanks for your time.

Kind regards,

Jelmer

Hello,thank your great project! I come again!
In the modules/RFRNet.py, the class RFRNet ,what is the work of self.RFRModule.att.att.att_scores_prev = None self.RFRModule.att.att.masks_prev = None?
Looking forward your reply

Hi Jin, thanks for sharing this great project!

I'm learning inpainting and would like to test it with your trained weights of face and places2
Is this possible to share them with me?

My email "ofirseo AT gmail.com"

Hey, thank you very much for sharing. I have a question, the verification set is not distinguished in the code, will it affect the final result?

@jingyuanli001 thanks for your excellent work! could you release your trained weight of places2?

Hi, I am very interested in your work. Tested the 256x256 image restoration effect is very good. How do I enter a larger picture for testing?

HI,Thank you for sharing this great project
But there was an error loading the model，The error is as follows：
‘’_ pickle.UnpicklingError : pickle data was truncated‘’
It may be your model upload error。
I hope you can give us an answer，Thank you

for i in range(6):
x2, m2 = self.Pconv21(x2, m2)
x2, m2 = self.Pconv22(x2, m2)
x2 = F.leaky_relu(self.bn2(x2), inplace = True)
x2 = self.RFRModule(x2, m2[:,0:1,:,:])
x2 = x2 * m2
About RFRNet code , I wonder why the result x2 after self.RFRModule needs to be carried out x2 = x2 * m2。This will obscure the newly generated pixels。Looking forward to your reply，thank you.

Thanks for author's contribution for this great work! But I got some trouble when training my own model.

RuntimeError occur in RFRNet_Smaller_Hole.py → RFRModule.forward(): When i is 1, 'h.shape' is 'torch.Size([6, 256, 68, 68])' and 'h_dict[enc_h_key].shape' is 'torch.Size([6, 128, 67, 67])', making 'h = torch.cat([h, h_dict[enc_h_key]], dim=1)' throws error 'invalid argument 0: Sizes of tensors must match except in dimension 1. Got 67 and 68 in dimension 2'.

Hope you can give me some ideas to fix this error. Thanks a lot!

I am fine-tuning the celeba model on a GTX 1080ti + Ryzen 7 2700X.

It looks like there is not a steady 100% GPU utilization (as I have with some other projects that I tested, mostly tensorflow):

Any reason why the GPU is not at 100% for the entire training process? Any way to improve this?

It takes 97 seconds for 50 iterations, which I think is a lot.

I am using random masks (mask mode 1), could that be the issue?

Thanks for your contributions.
'For CelebA Dataset, train the model for 350,000 iterations and finetune for 150,000 iterations. (500,000 in total)'
I want to know what's the meaning of finetune in the same dataset.
According to the code, when finetuning, parameters of BatchNorm2d will be fixed and the lr will change from 2e-4 to 5e-5.
Do I ignore some other operations ?

Hi, thanks for sharing! Great work!

What steps are necessary to make a 512 pixel square image model and train and eval on that?

Thanks!

First of all, thanks for your great work!
I got a question when looking through your code in Attention.py, where line 39 is as follows:

conv_result = F.avg_pool2d(conv_result, 3, 1, padding = 1)*9

this corresponds to the equation (5) in your paper:

My question is, the 3*3 average pooling itself is capable of getting the averaged results, why multiplied by 9 is needed?

Hope to hear your reply : )

Hello, why does the loss increase when I load the saved parameters after stopping the training and continue training?