# Inpainting ### *class* deepinv.physics.Inpainting(img_size, mask=None, pixelwise=True, device='cpu', rng=None, \*\*kwargs) Bases: [`DecomposablePhysics`](https://deepinv.org/api/stubs/deepinv.physics.DecomposablePhysics.html.md#deepinv.physics.DecomposablePhysics) Inpainting forward operator, keeps a subset of entries. The operator is described by the diagonal matrix $$ A = \text{diag}(m) \in \mathbb{R}^{n\times n} $$ where $m$ is a binary mask with $n$ entries. This operator is linear and has a trivial SVD decomposition, which allows for fast computation of the pseudo-inverse and proximal operator. An existing operator can be loaded from a saved `.pth` file via `self.load_state_dict(save_path)`, in a similar fashion to [`torch.nn.Module`](https://docs.pytorch.org/docs/stable/generated/torch.nn.Module.html#torch.nn.Module). Masks can also be created on-the-fly using mask generators such as [`BernoulliSplittingMaskGenerator`](https://deepinv.org/api/stubs/deepinv.physics.generator.BernoulliSplittingMaskGenerator.html.md#deepinv.physics.generator.BernoulliSplittingMaskGenerator), see example below. * **Parameters:** * **mask** ([*torch.Tensor*](https://docs.pytorch.org/docs/stable/tensors.html#torch.Tensor) *,* [*float*](https://docs.python.org/3.12/library/functions.html#float)) – If the input is a float, the entries of the mask will be sampled from a bernoulli distribution with probability equal to `mask`. If the input is a [`torch.Tensor`](https://docs.pytorch.org/docs/stable/tensors.html#torch.Tensor) matching `img_size`, the mask will be set to this tensor. If `mask` is [`torch.Tensor`](https://docs.pytorch.org/docs/stable/tensors.html#torch.Tensor), it must be shape that is broadcastable to input shape and will be broadcast during forward call. If `None`, it must be set during forward pass or using `update` method. * **img_size** ([*tuple*](https://docs.python.org/3.12/library/stdtypes.html#tuple)) – size of the input images without batch dimension e.g. of shape `(C, H, W)` or `(C, M)` or `(M,)`. * **device** ([*torch.device*](https://docs.pytorch.org/docs/stable/tensor_attributes.html#torch.device)) – gpu or cpu. * **pixelwise** ([*bool*](https://docs.python.org/3.12/library/functions.html#bool)) – Apply the mask in a pixelwise fashion, i.e., zero all channels in a given pixel simultaneously. If existing mask passed (i.e. mask is [`torch.Tensor`](https://docs.pytorch.org/docs/stable/tensors.html#torch.Tensor)), this has no effect. * **rng** ([*torch.Generator*](https://docs.pytorch.org/docs/stable/generated/torch.Generator.html#torch.Generator)) – a pseudorandom random number generator for the mask generation. Default to None.
* **Examples:** Inpainting operator using defined mask, removing the second column of a 3x3 image: ```pycon >>> from deepinv.physics import Inpainting >>> seed = torch.manual_seed(0) # Random seed for reproducibility >>> x = torch.randn(1, 1, 3, 3) # Define random 3x3 image >>> mask = torch.zeros(1, 3, 3) # Define empty mask >>> mask[:, 2, :] = 1 # Keeping last line only >>> physics = Inpainting(mask=mask, img_size=x.shape[1:]) >>> physics(x) tensor([[[[ 0.0000, -0.0000, -0.0000], [ 0.0000, -0.0000, -0.0000], [ 0.4033, 0.8380, -0.7193]]]]) ``` Inpainting operator using random mask, keeping 70% of the entries of a 3x3 image: ```pycon >>> from deepinv.physics import Inpainting >>> seed = torch.manual_seed(0) # Random seed for reproducibility >>> x = torch.randn(1, 1, 3, 3) # Define random 3x3 image >>> physics = Inpainting(mask=0.7, img_size=x.shape[1:]) >>> physics(x) tensor([[[[ 1.5410, -0.0000, -2.1788], [ 0.5684, -0.0000, -1.3986], [ 0.4033, 0.0000, -0.0000]]]]) ``` Generate random masks on-the-fly using mask generators: ```pycon >>> from deepinv.physics import Inpainting >>> from deepinv.physics.generator import BernoulliSplittingMaskGenerator >>> x = torch.randn(1, 1, 3, 3) # Define random 3x3 image >>> physics = Inpainting(img_size=x.shape[1:]) >>> gen = BernoulliSplittingMaskGenerator(x.shape[1:], split_ratio=0.7) >>> params = gen.step(batch_size=1, seed = 0) # Generate random mask >>> physics(x, **params) # Set mask on-the-fly tensor([[[[-0.4033, -0.0000, 0.1820], [-0.8567, 1.1006, -1.0712], [ 0.1227, -0.0000, 0.3731]]]]) >>> physics.update(**params) # Alternatively update mask before forward call >>> physics(x) tensor([[[[-0.4033, -0.0000, 0.1820], [-0.8567, 1.1006, -1.0712], [ 0.1227, -0.0000, 0.3731]]]]) ```
* **Used in benchmarks:** - [DIV2K Inpainting easy](https://deepinv.org/auto_benchmarks/div2k_inpainting_easy.html.md#div2k-inpainting-easy) #### \_\_mul_\_(other) Concatenates two forward operators $A = A_1\circ A_2$ via the mul operation If the second operator is an Inpainting or MRI operator, the masks are multiplied element-wise, otherwise the default implementation of LinearPhysics is used (see [`deepinv.physics.LinearPhysics.__mul__()`](https://deepinv.org/api/stubs/deepinv.physics.LinearPhysics.html.md#deepinv.physics.LinearPhysics.__mul__)). * **Parameters:** **other** ([*deepinv.physics.Physics*](https://deepinv.org/api/stubs/deepinv.physics.Physics.html.md#deepinv.physics.Physics)) – Physics operator $A_2$ * **Returns:** ([`deepinv.physics.Physics`](https://deepinv.org/api/stubs/deepinv.physics.Physics.html.md#deepinv.physics.Physics)) concatenated operator * **Return type:** [*Physics*](https://deepinv.org/api/stubs/deepinv.physics.Physics.html.md#deepinv.physics.Physics) #### noise(x, \*\*kwargs) Incorporates noise into the measurements $\tilde{y} = N(y)$ * **Parameters:** **x** ([*torch.Tensor*](https://docs.pytorch.org/docs/stable/tensors.html#torch.Tensor)) – clean measurements * **Return torch.Tensor:** noisy measurements * **Return type:** [*Tensor*](https://docs.pytorch.org/docs/stable/tensors.html#torch.Tensor) ## Examples using `Inpainting`:
![](auto_examples/basics/images/thumb/sphx_glr_demo_custom_dataset_thumb.png) [Bring your own dataset](https://deepinv.org/auto_examples/basics/demo_custom_dataset.html.md)
Bring your own dataset
![](auto_examples/basics/images/thumb/sphx_glr_demo_custom_optim_thumb.png) [Use iterative reconstruction algorithms](https://deepinv.org/auto_examples/basics/demo_custom_optim.html.md)
Use iterative reconstruction algorithms
![](auto_examples/basics/images/thumb/sphx_glr_demo_quickstart_thumb.png) [5 minute quickstart tutorial](https://deepinv.org/auto_examples/basics/demo_quickstart.html.md)
5 minute quickstart tutorial
![](auto_examples/models/images/thumb/sphx_glr_demo_foundation_model_thumb.png) [Inference and fine-tune a foundation model](https://deepinv.org/auto_examples/models/demo_foundation_model.html.md)
Inference and fine-tune a foundation model
![](auto_examples/models/images/thumb/sphx_glr_demo_training_thumb.png) [Training a reconstruction model](https://deepinv.org/auto_examples/models/demo_training.html.md)
Training a reconstruction model
![](auto_examples/optimization/images/thumb/sphx_glr_demo_dip_thumb.png) [Reconstructing an image using the deep image prior.](https://deepinv.org/auto_examples/optimization/demo_dip.html.md)
Reconstructing an image using the deep image prior.
![](auto_examples/optimization/images/thumb/sphx_glr_demo_epll_thumb.png) [Expected Patch Log Likelihood (EPLL) for Denoising and Inpainting](https://deepinv.org/auto_examples/optimization/demo_epll.html.md)
Expected Patch Log Likelihood (EPLL) for Denoising and Inpainting
![](auto_examples/optimization/images/thumb/sphx_glr_demo_multispectral_demosaicing_thumb.png) [Multispectral demosaicing from raw sensor data](https://deepinv.org/auto_examples/optimization/demo_multispectral_demosaicing.html.md)
Multispectral demosaicing from raw sensor data
![](auto_examples/optimization/images/thumb/sphx_glr_demo_wavelet_prior_thumb.png) [Image inpainting with wavelet prior](https://deepinv.org/auto_examples/optimization/demo_wavelet_prior.html.md)
Image inpainting with wavelet prior
![](auto_examples/physics/images/thumb/sphx_glr_demo_physics_tour_thumb.png) [Tour of forward sensing operators](https://deepinv.org/auto_examples/physics/demo_physics_tour.html.md)
Tour of forward sensing operators
![](auto_examples/plug-and-play/images/thumb/sphx_glr_demo_PnP_multiscale_thumb.png) [Multi-scale Plug-and-Play for Inpainting](https://deepinv.org/auto_examples/plug-and-play/demo_PnP_multiscale.html.md)
Multi-scale Plug-and-Play for Inpainting
![](auto_examples/sampling/images/thumb/sphx_glr_demo_ddrm_thumb.png) [Image reconstruction with a diffusion model](https://deepinv.org/auto_examples/sampling/demo_ddrm.html.md)
Image reconstruction with a diffusion model
![](auto_examples/sampling/images/thumb/sphx_glr_demo_diffpir_thumb.png) [Implementing DiffPIR](https://deepinv.org/auto_examples/sampling/demo_diffpir.html.md)
Implementing DiffPIR
![](auto_examples/sampling/images/thumb/sphx_glr_demo_diffusers_thumb.png) [Using state-of-the-art diffusion models from HuggingFace Diffusers with DeepInverse](https://deepinv.org/auto_examples/sampling/demo_diffusers.html.md)
Using state-of-the-art diffusion models from HuggingFace Diffusers with DeepInverse
![](auto_examples/sampling/images/thumb/sphx_glr_demo_diffusion_sde_thumb.png) [Building your diffusion posterior sampling method using SDEs](https://deepinv.org/auto_examples/sampling/demo_diffusion_sde.html.md)
Building your diffusion posterior sampling method using SDEs
![](auto_examples/sampling/images/thumb/sphx_glr_demo_dps_thumb.png) [DPS – Posterior Sampling with Diffusion Models](https://deepinv.org/auto_examples/sampling/demo_dps.html.md)
DPS -- Posterior Sampling with Diffusion Models
![](auto_examples/sampling/images/thumb/sphx_glr_demo_flow_matching_thumb.png) [Flow-Matching for posterior sampling and unconditional generation](https://deepinv.org/auto_examples/sampling/demo_flow_matching.html.md)
Flow-Matching for posterior sampling and unconditional generation
![](auto_examples/sampling/images/thumb/sphx_glr_demo_noisy_data_fidelity_thumb.png) [Noisy data-fidelity terms for diffusion posterior sampling](https://deepinv.org/auto_examples/sampling/demo_noisy_data_fidelity.html.md)
Noisy data-fidelity terms for diffusion posterior sampling
![](auto_examples/sampling/images/thumb/sphx_glr_demo_sampling_thumb.png) [Uncertainty quantification with PnP-ULA.](https://deepinv.org/auto_examples/sampling/demo_sampling.html.md)
Uncertainty quantification with PnP-ULA.
![](auto_examples/self-supervised-learning/images/thumb/sphx_glr_demo_ei_transforms_thumb.png) [Image transformations for Equivariant Imaging](https://deepinv.org/auto_examples/self-supervised-learning/demo_ei_transforms.html.md)
Image transformations for Equivariant Imaging
![](auto_examples/self-supervised-learning/images/thumb/sphx_glr_demo_equivariant_splitting_thumb.png) [Self-supervised learning with Equivariant Splitting](https://deepinv.org/auto_examples/self-supervised-learning/demo_equivariant_splitting.html.md)
Self-supervised learning with Equivariant Splitting
![](auto_examples/self-supervised-learning/images/thumb/sphx_glr_demo_multioperator_imaging_thumb.png) [Self-supervised learning from incomplete measurements of multiple operators.](https://deepinv.org/auto_examples/self-supervised-learning/demo_multioperator_imaging.html.md)
Self-supervised learning from incomplete measurements of multiple operators.
![](auto_examples/unfolded/images/thumb/sphx_glr_demo_deal_thumb.png) [DEAL denoising and reconstruction](https://deepinv.org/auto_examples/unfolded/demo_deal.html.md)
DEAL denoising and reconstruction
![](auto_examples/unfolded/images/thumb/sphx_glr_demo_unfolded_constrained_LISTA_thumb.png) [Unfolded Chambolle-Pock for constrained image inpainting](https://deepinv.org/auto_examples/unfolded/demo_unfolded_constrained_LISTA.html.md)
Unfolded Chambolle-Pock for constrained image inpainting