Diffusion Posterior Proximal Sampling for Image Restoration

Hongjie Wu*,1,2, Linchao He*,3, Mingqin Zhang1,2, Dongdong Chen4, Kunming Luo5, Mengting Luo3, Ji-Zhe Zhou1,2, Hu Chen3, Jiancheng Lv†,1,2
1College of Computer Science, Sichuan University
2Engineering Research Center of Machine Learning and Industry Intelligence, Ministry of Education
3National Key Laboratory of Fundamental Science on Synthetic Vision, Sichuan University
4Heriot-Watt University
5Hong Kong University of Science and Technology
ACM Multimedia 2024 Oral
*Equal Contribution, †Corresponding Author
Paper Code arXiv

Image restoration examples of our proposed DPPS, where n denotes the number of candidate samples. It is obvious that DPPS produces images with better details and quality as n increases.

Abstract

Diffusion models have demonstrated remarkable efficacy in generating high-quality samples. Existing diffusion-based image restoration algorithms exploit pre-trained diffusion models to leverage data priors, yet they still preserve elements inherited from the unconditional generation paradigm. These strategies initiate the denoising process with pure white noise and incorporate random noise at each generative step, leading to over-smoothed results. In this paper, we introduce a refined paradigm for diffusion-based image restoration. Specifically, we opt for a sample consistent with the measurement identity at each generative step, exploiting the sampling selection as an avenue for output stability and enhancement. Besides, we start the restoration process with an initialization combined with the measurement signal, providing supplementary information to better align the generative process. Extensive experimental results and analyses validate the effectiveness of our proposed approach across diverse image restoration tasks.

Proposed Method

High-level illustration of our proposed DPPS (n=2). Existing methods employ random sampling from the predicted distribution, our approach takes multiple samples and selecting the one with better data consistency at each step.

DPPS extract multiple candidate samples from the predicted distribution and choose the one with the highest measurement consistency.

Experimental Results

Comparing different baselines with our proposed DPPS for solving noisy linear inverse problems. Row 1: SR (x4), Row 2: 80% inpainting, Row 3: Gaussian deblurring.

Visual results on details to demonstrate the efficacy of our proposed method and to explore the impact of n.

Results to show the robustness of our method across different random seeds (denoted by Ours1 and Ours2), in line with the claims made in the paper.

Acknowledgement

We thank the anonymous reviewers for their insightful comments and feedback. We extend our gratitude to Mengxi Xie from SJTU and Linrui Dai from CQU for their constructive suggestions on the figures, although we ultimately did not adopt them. This work was supported by the Fundamental Research Funds for the Central Universities (No.1082204112364), the National Nature Science Foundation of China (No.62106161), and the Sichuan University Luzhou Municipal Government Strategic Cooperation Project (No.2022CDLZ-8).

BibTeX

@inproceedings{wu2024diffusion,
  title={Diffusion Posterior Proximal Sampling for Image Restoration},
  author={Wu, Hongjie and He, Linchao and Zhang, Mingqin and Chen, Dongdong and Luo, Kunming and Luo, Mengting and Zhou, Ji-Zhe and Chen, Hu and Lv, Jiancheng},
  booktitle={Proceedings of the 32nd ACM International Conference on Multimedia},
  pages={214--223},
  year={2024}
}