ReFIR: Grounding Large Restoration Models with Retrieval Augmentation

Recent advances in diffusion-based Large Restoration Models (LRMs) have significantly improved photo-realistic image restoration by leveraging the internal knowledge embedded within model weights.

However, existing LRMs often suffer from the hallucination dilemma, i.e., producing incorrect contents or textures when dealing with severe degradations, due to their heavy reliance on limited internal knowledge.

In this paper, we propose an orthogonal solution called the Retrieval-augmented Framework for Image Restoration (ReFIR), which incorporates retrieved images as external knowledge to extend the knowledge boundary of existing LRMs in generating details faithful to the original scene.

Specifically, we first introduce the nearest neighbor lookup to retrieve content-relevant high-quality images as reference, after which we propose the cross-image injection to modify existing LRMs to utilize high-quality textures from retrieved images.

Thanks to the additional external knowledge, our ReFIR can well handle the hallucination challenge and facilitate faithfully results.

Extensive experiments demonstrate that ReFIR can achieve not only high-fidelity but also realistic restoration results.

Importantly, our ReFIR requires no training and is adaptable to various LRMs.

;Comment: Accepted by NeurIPS 2024