Hi, thanks for your meaningful work. However, when I reproduce PASD as your instruction in README, I notice the CPU memory keeps incereaing. Then my machine collapsed after 60k iterations, where the CPU memory usage goes to ~1T. Any ideas about this situation? Thanks for your any suggestion.
BTW, I employ the recomanded WebDataset + torch.Dataloader.

Hi,
Thanks so much for making this amazing model! Would you consider making it open source by adding a OSI-approved license (ie MIT/Apache 2.0/ISC)?
Thank you!

Hi! To reproduce your intersting work, i hope you can release your dataset which matches the formation defined in webdataset.py
thanks a lot!

How to solve it ?

Dear authors,
I have read about your PASD work,however, when I used the DIV2K dataset for training, I could not find the configuration file params_realesrgan.yml, and after checking, I found that there was no problem with the file name and path.

I want to train PASD on DIV2K_train_HR dataset. But I don't know how to change the code in webdatasets.py. Do I only need to change wds_urls into my dataset path? I already make my dataset a 'tar' file.

Hello! Is there any package installation guidance? I try to use pip install -r requirements.txt. However, the diffusers package with the latest one cannot find some function sometimes. Is there any specific version for diffusers used in the experiments of the paper? Thanks!

PASD is very nice job!
I was trying to make a personalize style(toonyou) photo, the original photo is 000020x2.png in the folder samples. However, I have tried toonyou.safetensors in civatai, with pasd_rrdb/checkpoint-100000, --use_personalized_model, others as default. I can not generalize the beautiful photo just like you paper show. Can you tell me what are the parameters in personalized_model?
Thank you!

I'll try it, but fail.

import cv2
import torch
from modelscope.outputs import OutputKeys
from modelscope.pipelines import pipeline
from modelscope.utils.constant import Tasks

input_location = 'http://public-vigen-video.oss-cn-shanghai.aliyuncs.com/robin/results/output_test_pasd/0fbc3855c7cfdc95.png'
prompt = ''
output_image_path = 'result.png'

input = {
    'image': input_location,
    'prompt': prompt,
    'upscale': 2,
    'fidelity_scale_fg': 1.0,
    'fidelity_scale_bg': 1.0,
    'use_personalized_model': True,
    'personalized_model_path': 'toonyou_beta3.safetensors'
}
pasd = pipeline(Tasks.image_super_resolution_pasd, model='damo/PASD_image_super_resolutions')
output = pasd(input)[OutputKeys.OUTPUT_IMG]
cv2.imwrite(output_image_path, output)
print('pipeline: the output image path is {}'.format(output_image_path))

where pasd_color

I want to train PASD on DIV2K_train_HR dataset. But when I change the wds_urls into wds_urls = "datasets/DIV2K_train_HR.tar".
This error occured.
I just changed the code in webdatasets.py : line 256.

Thank you for sharing your work. I want to know how to train a colorization model on my own dataset.

Hi, I would like to ask if there is a calculation code for PSNR, SSIM, FID, LPIPS, DISTS, MUSIQ and other related evaluation indicators, can you share it，thank you.

During training, the RAM keeps increasing. Is it a memory leak? But I can’t find where the problem is. Can anyone help me?

1. At first glance the data for PACA is coming after zero conv module as per the code, but in the article's "Figure 2: Architecture of the proposed pixel-aware stable diffusion (PASD) network", the bottom right yellow picture shows data for PACA coming from right before the zero conv with a split. Which one of this is effective or right?
2. In the same diagram, I don't see the unet noisy input being added to the controlnet data, but the code does. Which one is right.

The image generated by the model I trained at checkpoint-340000 is noisy, what's wrong with it

Do we only need to add added_prompt and negative_prompt during inference, not during training?

Output of this image

is

I wonder if my settings are incorrect or if the image degradation is too severe for super-resolution?

local variable 'validation_image' referenced before assignment

Hi. I made a colab notebook for inference.
I think PASD is underrated compared to other diffusion prior based models like DiffBIR.

Can you please post the details of the test dataset? I see there are no instructions specifically for the DIV2K valid dataset. I used the weights you posted to test on the DIV2K valid dataset posted by stablesr and found that the results were not the same.

Updated from latest changes and now get:

$ python test_pasd.py
C:\PASD\pipelines\pipeline_pasd.py:41: FutureWarning: Importing `DiffusionPipeline` or `ImagePipelineOutput` from diffusers.pipeline_utils is deprecated. Please import from diffusers.pipelines.pipeline_utils instead.
  from diffusers.pipeline_utils import DiffusionPipeline
C:\Python310\lib\site-packages\torchvision\transforms\functional_tensor.py:5: UserWarning: The torchvision.transforms.functional_tensor module is deprecated in 0.15 and will be **removed in 0.17**. Please don't rely on it. You probably just need to use APIs in torchvision.transforms.functional or in torchvision.transforms.v2.functional.
  warnings.warn(
clean, high-resolution, 8k
  0%|          | 0/20 [00:00<?, ?it/s]
too many values to unpack (expected 2)

测试时，output里没有生成图像

python test_pasd.py --use_personalized_model
/home/root1/anaconda3/envs/pasd/lib/python3.10/site-packages/diffusers/utils/outputs.py:63: UserWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead.
torch.utils._pytree._register_pytree_node(
/home/root1/anaconda3/envs/pasd/lib/python3.10/site-packages/diffusers/utils/outputs.py:63: UserWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead.
torch.utils._pytree._register_pytree_node(
/home/root1/anaconda3/envs/pasd/lib/python3.10/site-packages/diffusers/utils/outputs.py:63: UserWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead.
torch.utils._pytree._register_pytree_node(
/home/root1/anaconda3/envs/pasd/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: resume_download is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use force_download=True.
warnings.warn(
config.json: 4.52kB [00:00, 15.6MB/s]
INFO:root:Loaded coca_ViT-L-14 model config.
INFO:root:Loading pretrained coca_ViT-L-14 weights (mscoco_finetuned_laion2B-s13B-b90k).
a dog sitting in the grass with its tongue hanging out . clean, high-resolution, 8k
cannot fit 'int' into an index-sized integer

Hi, thanks for this excellent work! I would like to try on my own dataset (like hr_512, lr_128), could you please tell me which dataloader I should use to change to my own dataset path?

Thanks

Thanks for your impressive work, PASD. Now I encounter some issues while reproducing your Real-IR experiment results and would appreciate your assistance.

After training the model for 500k steps(as instructed in this repo), I tested the performance of different step model weights on the benchmark dataset, like 50k, 100k, 200k. However, the inference results of these weights I replicated are ALL NOT as impressive as yours. These are looked more blurry&noisy and lack sharpness. (FYI, I have included several comparative sample images from DRealSRx4 dataset, 128px -> 512px, in the attachment)

After code checking, I think the code is clean. Perhaps there's a discrepancy in the configuration of the degraded model parameters, i.e real-esrgan configure parameters? Or there are any magic training trick?

I would greatly appreciate any ideas or assistance you can provide! Thank you again!

Dear authors,
I have read about your PASD work, it mentioned PASD also support colorization task, may I ask whether the currently released PASD/PASD-light/PASD-RRDB including the colorization or not?
if not, may I ask will the colorization model be released? Many thanks.

thanks for help！

The colab notebook does upscale but doesn't add details to the image. I have checked same settings on the colab and the demospace, thedemospace does excelent job on adding details.
In the colab i get warning in the last cell, then it keeps working and gives me an upscale:

2024-01-26 10:42:39.581017: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:9261] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
2024-01-26 10:42:39.581065: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:607] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
2024-01-26 10:42:39.582556: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1515] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2024-01-26 10:42:39.590674: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2024-01-26 10:42:41.313011: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
/usr/local/lib/python3.10/dist-packages/torchvision/transforms/functional_tensor.py:5: UserWarning: The torchvision.transforms.functional_tensor module is deprecated in 0.15 and will be removed in 0.17. Please don't rely on it. You probably just need to use APIs in torchvision.transforms.functional or in torchvision.transforms.v2.functional.
warnings.warn(

when I inference 1024*1024 image , [Tiled VAE]: the input size is tiny and unnecessary to tile. [Tiled VAE]: Done in 12.476s, max VRAM alloc 5802.607 MB
But I want to use more gpu and fast than now , how to solve it ?
And inference once some model offload in gpu

Hi, could you share the ControlNet model producing the results in Fig. 1?

I want to test pasd_light using python test_pasd.py --use_pasd_light .But I encountered this error:

Do you have any suggestion? Looking forward to ur reply~

I test according to readme.md and report an error.

"cusolver error: CUSOLVER_STATUS_EXECUTION_FAILED, when calling cusolverDnSgetrf( handle, m, n, dA, ldda, static_cast<float*>(dataPtr.get()), ipiv, info). This error may appear if the input matrix contains NaN."

There is a problem with "latents = self.scheduler.step(noise_pred, t, latents, **extra_step_kwargs, return_dict=False)[0]" in line 1151, pipelines/pipeline_pasd.py.

I hope you can give me some guidance, thank you very much!

The text information obtained by image classification and target detection are all frogs. The text information obtained by the BLIP network also contains frogs. What is the role of repeated text?

Hi, authors,

Congrats for the nice work.

I wonder what is the model/config for colorization?

Thx a lot

May I ask which datasets you have used? In the paper, you mentioned using div2k and flickr2k, but I see that there are also datasets such as FFHQ on the GitHub you published.

when I download pre-trained models pasd_light and put them into runs/. And I download SD1.5 models v1-5-pruned-emaonly.ckpt and put them into checkpoints/stable-diffusion-v1-5. I run python test_pasd.py --use_pasd_light there is an error :
OSError: Error no file named pytorch_model.bin, model.safetensors, tf_model.h5, model.ckpt.index or flax_model.msgpack found in directory checkpoints/stable-diffusion-v1-5.
Am I missing something that needs to be downloaded?

Dear authors,
I have read about your PASD work,and I used the DIV2K dataset for training,due to the limitation of GPU (48GB of graphics memory), I reduced the epochs, batch size, and steps, but the training progress bar did not move, and in the end, the model weights were saved.

can you help me？

Very nice job! When I want to train the model, I found that the training dataset URL is broken. May I ask for you updating the training dataset URL. Thank you very much.

Usually on darker areas, there is some noise like this:

Is there a way to reduce that noise? Or add something to the code to denoise it? Thank you🙏

Hi~, can I ask for some detail about the model.
In Figure 2 of the paper, it is shown that PACA only exists in the up-sample blocks of U-Net.
Is there any specific motivation behind this design?
Looking forward to your reply.
Thanks very much
@yangxy

thanks for your work and it is really interesting! However, while reading your code I can't make it clear that which part of the model is responsible for the degradation removal work.
in line 927 of train_pasd.py , you calculated F.l1_loss(pixel_values.float(), controlnet_cond_mid.float(), reduction="mean")
so you mean controlnet_cond_mid is the denoised image for diffusion model? I'm not sure if I understood your idea

Hi, authors.

Thx for your nice open-source work, I wonder what are the rrdb/light?