best fid score? about arldm HOT 47 OPEN

@KyonP Hi, our config is the same as https://github.com/xichenpan/ARLDM/blob/main/config.yaml
However, as discussed in this thread, it is quite weird the implementation in Pytorch Lighting cannot fully reproduce the result. I spend a few hours finding the difference between the two version. However, I have mad too much modification so that I cannot distinguish what cause the problem. You are welcome to refer to our original implementation https://github.com/xichenpan/ARLDM/tree/a24e2e94332eb86fcc071abb83aaf341006aa622, if you find some bug in our pytorch lightning implementation, I am more than happy to know!
Besides, we plan to release the trained weight so that you can use it to inference.

from arldm.

@yangsenwxy Hi, could you please provide some visual samples? and also the training config like how many epochs, freeze clip/blip/resnet or not.

from arldm.

btw, in our experiment, we just use the ckpt in 50 epochs, cause the loss is irrelevant with the fid score.

from arldm.

can I have you wechat?I tell you by email? [email protected]

from arldm.

This is my config

from arldm.

can I have you wechat?I tell you by email? [email protected]

@yangsenwxy Yeah, I am happy to chat. But I prefer to discuss in this thread because this is a new implementation and we haven't train a model using this repo. If we can figure out the problem, it may help a lot~

from arldm.

This is my config

It seems the init_lr is too large, you may try our config of 1e-5. Also, could you provide your sample config, like scheduler and steps?

from arldm.

I will give you some exsample

from arldm.

I will give you some exsample

You may follow the config given in https://github.com/Flash-321/ARLDM/blob/main/config.yaml (which is the config for the report performance.) The ddim scheduler with 50 steps provide a 2-3 higher FID score result. We suggest using ddim-6-250 or pndm-7.5-50 (this is faster and provide a 1-2 higher FID score)

from arldm.

I will give you some exsample

Thanks a lot! Only the generated images is enough (caption is not related to FID score)

from arldm.

链接：https://pan.baidu.com/s/1s4WU9xdS7Qn_XO-O2kvwaA
提取码：n1dl
--来自百度网盘超级会员V5的分享

from arldm.

here

from arldm.

here

Copy! So I think it is a training issue, since the generated visual stories are unacceptable.

from arldm.

@yangsenwxy Have you ever try to train the model for 50 epochs using a init_lr of 1e-5?

from arldm.

I will try it

from arldm.

I will try it

@yangsenwxy Thanks! I have tried using a large lr, but it produce a bad performance when training for many epochs. You can also try to train the model for 5 epochs using a lr of 1e-4 and see if it works. I tried this setting in my early experiment and it can also generate reasonable results. Feel free to at me in this thread if there is an update or further issue!

from arldm.

Ok, let me try, do you mean smaller learning rate, smaller FID?

from arldm.

Ok, let me try, do you mean smaller learning rate, smaller FID?

@yangsenwxy Not actually, but large learning rate do not work well when you train the model really long.

from arldm.

@Flash-321 I used your exact same settings for the last 50 epoch and ran out a FID score of 25

from arldm.

@yangsenwxy yeah, and do you also use the default setting during sampling?

from arldm.

also, may I see your learning rate curve to find out if the scheduler work correctly?

from arldm.

OK，wait for me for half an hour

from arldm.

Sure!

from arldm.

@yangsenwxy Really sorry about a bug in our code. I just check our dataset implementation, and found that during immigranting to Pytorch Lightning code, we forget to add the normalization for training data.
Here is our internal implementation:
https://github.com/Flash-321/ARLDM/blob/a24e2e94332eb86fcc071abb83aaf341006aa622/ARLDM.py#L47-L52
Here is the pytorch lightning implementation in current branch:
https://github.com/Flash-321/ARLDM/blob/eb907e3717ac20f82dfba8e67fd55d95127de098/datasets/flintstones.py#L26-L31
This will do harm to the PTMs since Stable Diffusion is trained on these normalized images.

My sincere apologize for this issue. The FID score should be further improved after fixing this bug.

from arldm.

@yangsenwxy Really sorry about a bug in our code. I just check our dataset implementation, and found that during immigranting to Pytorch Lightning code, we forget to add the normalization for training data. Here is our internal implementation:

https://github.com/Flash-321/ARLDM/blob/a24e2e94332eb86fcc071abb83aaf341006aa622/ARLDM.py#L47-L52

Here is the pytorch lightning implementation in current branch:
https://github.com/Flash-321/ARLDM/blob/eb907e3717ac20f82dfba8e67fd55d95127de098/datasets/flintstones.py#L26-L31

This will do harm to the PTMs since Stable Diffusion is trained on these normalized images.
My sincere apologize for this issue. The FID score should be further improved after fixing this bug.

And this bug only happends in FlintstonesSV dataset, if you have done experiments in other datasets, the performance was not affected.

from arldm.

I discovered this bug a long time ago, and it has been corrected during the original training

from arldm.

@yangsenwxy yeah, and do you also use the default setting during sampling?

Yes

from arldm.

also, may I see your learning rate curve to find out if the scheduler work correctly?

from arldm.

@yangsenwxy

also, may I see your learning rate curve to find out if the scheduler work correctly?

Thanks, it looks correct.

from arldm.

Now,FID is 25.069521856381982

from arldm.

Now,FID is 25.069521856381982

Got it! Could you please post 1-2 visual samples on this thread? There are also several differences between the two implementation. Like in this version, we use stable diffusion v1.5 instead of v1.4, but I think it doesn't matter. We also use a gradient clip in our training process, I guess this may be a factor
https://github.com/Flash-321/ARLDM/blob/a24e2e94332eb86fcc071abb83aaf341006aa622/config/config_flintstones.json#L51-L66
maybe you can go through the training log and see if the gradient is too large in some step? If so, I guess adding this config may help:

trainer = Trainer(gradient_clip_val=1.0)

You can also check our original internal implementation (https://github.com/Flash-321/ARLDM/tree/a24e2e94332eb86fcc071abb83aaf341006aa622) to find some difference, we can discuss them in this thread.

from arldm.

Here

from arldm.

from arldm.

Here ,some visual samples

from arldm.

@yangsenwxy Copy, your experiments help a lot, thanks for that! It looks like a training issue, the val loss should be around 0.11, and the visual samples are not reasonable and coherent.
I recommend add clip_norm and train for 5 epochs in a learing rate of 1e-4 (this can help reduce traning cost.), so that we can find out whether it is due to the large gradient.

from arldm.

OK

from arldm.

@Flash-321 I have added clip_norm as you suggested, but the FID is 24. 24.87433417204761

from arldm.

loss is here

from arldm.

from arldm.

@yangsenwxy Hi, thanks for your feedback, it seems the conditioning part is ruined, since each frame is not coherent. I will check the implementation soon and also ask my mentor to release the ckpt and training log for reference.

from arldm.

hello. I also have a difficult time reproducing your best-performing results on Pororo.

Can you share some config settings?

from arldm.

@xichenpan
I managed to achieve an FID score of around 18 with PL version of your code. It looks good enough for me right now 👍

I am looking forward to having your pre-trained weights.

BTW, is there any chance you will improve the inference speed of this code? it is very time-consuming.

from arldm.

@KyonP Great! I just asked my mentor, and he told me the release request have been approved by Alibaba, we will provide the pre trained weight this week!

from arldm.

Hey, I found the same issue. @xichenpan Did you try to save the images first and then evaluate the FID? Or you only use the FID calculation in the "main.py" code? Cause I found the Inception Network in the code seems to lose ".eval()", which would norm the features and get a lower FID.

from arldm.

@xichenpan Hi, I am having a hard time finding the released weight, and I am trying to do some follow-ups of your work. If you haven't upload them yet, can you share them via google drive or maybe other platforms?

from arldm.

@xichenpan, Are your checkpoints ready ? If possible, will you be ready to share them. Can you explain, how to learn ARLDM with one available CUDA index. How to beat CUDA out of memory error using CLIP, BLIP, RESNET freezing or other methodics.

from arldm.

Can somebody (who made training) share checkpoints, because I have CUDA out of memory, training the model.

from arldm.