#### NOTE New to DeepInverse? Get started with the basics with the [5 minute quickstart tutorial](https://deepinv.org/auto_examples/basics/demo_quickstart.html.md#sphx-glr-auto-examples-basics-demo-quickstart-py).. # Using state-of-the-art diffusion models from HuggingFace Diffusers with DeepInverse This demo shows you how to use our wrapper [`deepinv.models.DiffusersDenoiserWrapper`](https://deepinv.org/api/stubs/deepinv.models.DiffusersDenoiserWrapper.html.md#deepinv.models.DiffusersDenoiserWrapper) to turn any SOTA models from the HuggingFace Hub to an image denoiser. It also can be used to perform unconditional image generation or for posterior sampling. See more about the [diffusers pipeline](https://huggingface.co/docs/diffusers/index) and our posterior sampling [user guide](https://deepinv.org/auto_examples/sampling/demo_diffusion_sde.html.md#sphx-glr-auto-examples-sampling-demo-diffusion-sde-py). #### NOTE This example requires the `diffusers` and `transformers` package. You can install it via `pip install diffusers transformers`. ```Python import torch import deepinv as dinv from deepinv.models.wrapper import DiffusersDenoiserWrapper device = dinv.utils.get_device() dtype = torch.float32 figsize = 2.5 from deepinv.sampling import ( PosteriorDiffusion, VarianceExplodingDiffusion, AncestralSolver, ) from deepinv.optim import ZeroFidelity ``` ```none Selected GPU 0 with 24085.25 MiB free memory ``` --- Let us first load a pretrained diffusion model from the HuggingFace Hub. Here, we use the `google/ddpm-ema-celebahq-256` model. This model is trained on 256x256 of CelebA dataset using the DDPM scheduler. ```Python # We can wrap any diffusers model as a DeepInv denoiser using one line of code: denoiser = DiffusersDenoiserWrapper( mode_id="google/ddpm-ema-celebahq-256", device=device ) # Load an example image x = dinv.utils.load_example( "celeba_example2.jpg", img_size=256, resize_mode="resize", ).to(device) # Add noise and test the denoiser sigma = 0.1 x_noisy = x + sigma * torch.randn_like(x) with torch.no_grad(): x_denoised = denoiser(x_noisy, sigma=sigma) dinv.utils.plot( [x, x_noisy, x_denoised], figsize=(figsize * 3, figsize), titles=["Original image", "Noisy image", "Denoised image"], ) ``` ```none /local/jtachell/deepinv/deepinv/.pixi/envs/docs/lib/python3.12/site-packages/diffusers/utils/deprecation_utils.py:23: FutureWarning: `torch_dtype` is deprecated and will be removed in version 1.0.0. Please use `dtype` instead. deprecate("torch_dtype", "1.0.0", _TORCH_DTYPE_DEPRECATION_MESSAGE) Cannot initialize model with low cpu memory usage because `accelerate` was not found in the environment. Defaulting to `low_cpu_mem_usage=False`. It is strongly recommended to install `accelerate` for faster and less memory-intense model loading. You can do so with: ``` pip install accelerate ``` . Loading pipeline components...: 0%| | 0/2 [00:00 --- It is also possible to use the wrapped model for unconditional image generation. The model was trained with DDPM scheduler, however we can use it with any SDE provided in DeepInv. Here, we use the Variance Exploding SDE with Euler solver for sampling. ```Python num_steps = 50 rng = torch.Generator(device) sde = VarianceExplodingDiffusion( device=device, dtype=dtype, ) solver = AncestralSolver(t_start=sde.T, t_end=0.0, num_steps=num_steps, rng=rng) model = PosteriorDiffusion( data_fidelity=ZeroFidelity(), sde=sde, denoiser=denoiser, solver=solver, dtype=dtype, device=device, verbose=True, ) sample, trajectory = model( y=None, physics=None, x_init=(1, 3, 256, 256), seed=42, get_trajectory=True, ) dinv.utils.plot( sample, titles="Unconditional generation", figsize=(figsize, figsize), ) ``` ```none 0%| | 0/49 [00:00 --- Similar to other denoisers in DeepInv, the wrapped diffusers model can be used for posterior sampling. Below we use the same VE-SDE for posterior sampling in an inpainting problem. ```Python # Initialize the physics mask = torch.ones_like(x) mask[..., 70:150, 120:180] = 0 physics = dinv.physics.Inpainting( mask=mask, img_size=x.shape[1:], device=device, noise_model=dinv.physics.GaussianNoise(0.05), ) y = physics(x) ``` ```Python from deepinv.sampling import DPSDataFidelity model = PosteriorDiffusion( data_fidelity=DPSDataFidelity(denoiser=denoiser, weight=1.0, guidance="annealed"), denoiser=denoiser, sde=sde, solver=solver, dtype=dtype, device=device, verbose=True, ) ``` ```Python posterior_sample = model( y=y, physics=physics, x_init=(1, 3, 256, 256), seed=15, ) dinv.utils.plot( [x, y, posterior_sample], titles=["Original image", "Measurement", "Posterior sample"], figsize=(figsize * 3, figsize), ) ``` ```none 0%| | 0/49 [00:00 [`Download Jupyter notebook: demo_diffusers.ipynb`](https://deepinv.org/auto_examples/sampling/_downloads/873cb38fbf05f1e9edd3645a4789a8e8/demo_diffusers.ipynb) [`Download Python source code: demo_diffusers.py`](https://deepinv.org/auto_examples/sampling/_downloads/6285efda42c6303549a0b3b27df44f44/demo_diffusers.py) [`Download zipped: demo_diffusers.zip`](https://deepinv.org/auto_examples/sampling/_downloads/f610ff1cea81f80a082154b749a021a8/demo_diffusers.zip) [Gallery generated by Sphinx-Gallery](https://sphinx-gallery.github.io)