#### NOTE New to DeepInverse? Get started with the basics with the [5 minute quickstart tutorial](https://deepinv.org/auto_examples/basics/demo_quickstart.html.md#sphx-glr-auto-examples-basics-demo-quickstart-py).. # Bring your own dataset This example shows how to use DeepInverse with your own dataset. A dataset in DeepInverse can consist of optional ground-truth images `x`, measurements `y`, or [physics parameters](https://deepinv.org/user_guide/physics/intro.html.md#parameter-dependent-operators) `params`, or any combination of these. See [datasets user guide](https://deepinv.org/user_guide/training/datasets.html.md#datasets) for the formats we expect data to be returned in for compatibility with DeepInverse (e.g., to be used with [`deepinv.Trainer`](https://deepinv.org/api/stubs/deepinv.Trainer.html.md#deepinv.Trainer)). DeepInverse provides multiple ways of bringing your own dataset. This example has two parts: firstly how to load images/data into a dataset, and secondly how to use this dataset with DeepInverse. ```Python import deepinv as dinv import torch from torch.utils.data import DataLoader from torchvision.transforms import ToTensor ``` ## Part 1: Loading data into a dataset ### You have a folder of ground truth images Here we imagine we have a folder with one ground truth image of a butterfly. #### TIP [`deepinv.datasets.ImageFolder`](https://deepinv.org/api/stubs/deepinv.datasets.ImageFolder.html.md#deepinv.datasets.ImageFolder) can load any type of data (e.g. MRI, CT, etc.) by passing in a custom `loader` function and `transform`. ```Python DATA_DIR = dinv.utils.get_cache_home() / "demo_custom_dataset" dinv.utils.download_example("butterfly.png", DATA_DIR / "GT") dataset1 = dinv.datasets.ImageFolder(DATA_DIR / "GT", transform=ToTensor()) # Load one image from dataset x = next(iter(DataLoader(dataset1))) dinv.utils.plot({"x": x}) ``` ### You have a folder of paired ground truth and measurements Now imagine we have a ground truth folder with a butterfly, and a measurements folder with a masked butterfly. ```Python dinv.utils.download_example("butterfly_masked.png", DATA_DIR / "Measurements") dataset2 = dinv.datasets.ImageFolder( DATA_DIR, x_path="GT/*.png", y_path="Measurements/*.png", transform=ToTensor() ) x, y = next(iter(DataLoader(dataset2))) dinv.utils.plot({"x": x, "y": y}) ``` #### NOTE If you’re loading measurements which have randomly varying `params`, your dataset must return tuples `(x, y, params)` so that the physics is modified accordingly every image. We provide a convenience argument `ImageFolder(estimate_params=...)` to help you estimate these `params` on the fly. ### You have a folder of only measurements Imagine you have no ground truth, only measurements. Then `x` should be loaded in as NaN: ```Python dataset3 = dinv.datasets.ImageFolder( DATA_DIR, y_path="Measurements/*.png", transform=ToTensor() ) x, y = next(iter(DataLoader(dataset3))) print(x) ``` ```none tensor([nan], dtype=torch.float64) ``` ### You already have tensors Sometimes you might already have tensor(s). You can construct a dataset using [`deepinv.datasets.TensorDataset`](https://deepinv.org/api/stubs/deepinv.datasets.TensorDataset.html.md#deepinv.datasets.TensorDataset), for example here an unsupervised dataset containing just a single measurement (and will be loaded in as a tuple `(nan, y)`): ```Python y = dinv.utils.load_example("butterfly_masked.png") dataset4 = dinv.datasets.TensorDataset(y=y) x, y = next(iter(DataLoader(dataset4))) print(x) ``` ```none tensor([nan], dtype=torch.float64) ``` ### You already have a PyTorch dataset Say you already have your own PyTorch dataset: ```Python class MyDataset(torch.utils.data.Dataset): def __len__(self): return 1 def __getitem__(self, i): # Returns (x, y, params) return torch.zeros(1), torch.zeros(1), {"mask": torch.zeros(1)} dataset5 = MyDataset() ``` You should check that your dataset is compatible using [`deepinv.datasets.check_dataset()`](https://deepinv.org/api/stubs/deepinv.datasets.check_dataset.html.md#deepinv.datasets.check_dataset) (alternatively inherit from [`deepinv.datasets.ImageDataset`](https://deepinv.org/api/stubs/deepinv.datasets.ImageDataset.html.md#deepinv.datasets.ImageDataset) and use `self.check_dataset()`): ```Python dinv.datasets.check_dataset(dataset5) ``` ```none /local/jtachell/deepinv/deepinv/examples/basics/demo_custom_dataset.py:117: DeprecationWarning: The tuple format for dataset outputs is deprecated and will be removed in a future version.It is recommended to implement your dataset to return a dict with keys 'x', 'y', and 'params' instead of a tuple. dinv.datasets.check_dataset(dataset5) ``` ## Part 2: Using your dataset with DeepInverse Say you have a DeepInverse problem already set up: ```Python device = dinv.utils.get_device() physics = dinv.physics.Inpainting(img_size=(3, 256, 256)) model = dinv.models.RAM(pretrained=True, device=device) ``` ```none Selected GPU 0 with 28678.0 MiB free memory ``` If your dataset already returns measurements in the form `(x, y)` or `(x, y, params)`, you can directly test with it. Our physics does not yet know the `params` (here, the inpainting mask). Since it is fixed across the dataset, we can define it manually by estimating it from y: #### NOTE If you’re loading measurements which have randomly varying `params`, your dataset must return tuples `(x, y, params)` so that the physics is modified accordingly every image. ```Python params = {"mask": (dataset2[0][1].to(device) != 0).float()} physics.update(**params) dinv.test(model, DataLoader(dataset2), physics, plot_images=True, device=device) ``` ```none /local/jtachell/deepinv/deepinv/deepinv/training/trainer.py:593: UserWarning: Update progress bar frequency of 1 may slow down training on GPU. Consider setting freq_update_progress_bar > 1. warnings.warn( 0%| | 0/1 [00:00 Even if the dataset doesn’t have ground truth, no-reference metrics can be used. ```Python metrics = [dinv.metric.SharpnessIndex()] # dinv.metric.NIQE(device=device) dinv.test( model, DataLoader(dataset3), physics, plot_images=True, metrics=metrics, device=device, ) ``` ```none /local/jtachell/deepinv/deepinv/deepinv/training/trainer.py:593: UserWarning: Update progress bar frequency of 1 may slow down training on GPU. Consider setting freq_update_progress_bar > 1. warnings.warn( 0%| | 0/1 [00:00 ## Generating measurements If your dataset returns only ground-truth `x`, you can generate a dataset of measurements using [`deepinv.datasets.generate_dataset()`](https://deepinv.org/api/stubs/deepinv.datasets.generate_dataset.html.md#deepinv.datasets.generate_dataset): ```Python path = dinv.datasets.generate_dataset( dataset1, physics, save_dir=DATA_DIR / "measurements", device=device ) dinv.test( model, DataLoader(dinv.datasets.HDF5Dataset(path, use_dict_output=True)), physics, plot_images=True, device=device, ) ``` ```none /local/jtachell/deepinv/deepinv/deepinv/datasets/datagenerator.py:624: UserWarning: Dataset /local/jtachell/.cache/deepinv/demo_custom_dataset/measurements/dinv_dataset0.h5 already exists, this will close and overwrite the previous dataset. warn( Dataset has been saved at /local/jtachell/.cache/deepinv/demo_custom_dataset/measurements/dinv_dataset0.h5 /local/jtachell/deepinv/deepinv/deepinv/datasets/datagenerator.py:181: UserWarning: Neither 'split' nor 'train' parameters are provided. Defaulting to loading the 'train' split. This behaviour is likely to change in future versions of the library. warn( /local/jtachell/deepinv/deepinv/deepinv/training/trainer.py:593: UserWarning: Update progress bar frequency of 1 may slow down training on GPU. Consider setting freq_update_progress_bar > 1. warnings.warn( 0%| | 0/1 [00:00 #### TIP Pass in a [physics generator](https://deepinv.org/user_guide/physics/intro.html.md#physics-generators) to simulate random physics and then use `load_physics_generator_params=True` to load these `params` alongside the data during testing. If you don’t want to generate a dataset offline, you can also generate measurements online (“on-the-fly”) during testing or training: ```Python dinv.test( model, DataLoader(dataset1), physics, plot_images=True, device=device, online_measurements=True, ) ``` ```none /local/jtachell/deepinv/deepinv/deepinv/training/trainer.py:593: UserWarning: Update progress bar frequency of 1 may slow down training on GPU. Consider setting freq_update_progress_bar > 1. warnings.warn( 0%| | 0/1 [00:00 🎉 Well done, you now know how to use your own dataset with DeepInverse! ### What’s next? * Check out [the example on how to test a state-of-the-art general pretrained model](https://deepinv.org/auto_examples/basics/demo_pretrained_model.html.md#sphx-glr-auto-examples-basics-demo-pretrained-model-py) with your new dataset. * Check out the [example on how to fine-tune a foundation model](https://deepinv.org/auto_examples/models/demo_foundation_model.html.md#sphx-glr-auto-examples-models-demo-foundation-model-py) to your own data. * Check out the [example on how to train a reconstruction model](https://deepinv.org/auto_examples/models/demo_training.html.md#sphx-glr-auto-examples-models-demo-training-py) with your dataset. * Advanced: how to [stream or download a dataset from HuggingFace](https://deepinv.org/auto_examples/external-libraries/demo_hf_dataset.html.md#sphx-glr-auto-examples-external-libraries-demo-hf-dataset-py). **Total running time of the script:** (0 minutes 2.174 seconds) [`Download Jupyter notebook: demo_custom_dataset.ipynb`](https://deepinv.org/auto_examples/basics/_downloads/5611eb7a5e4c3863d7158fef25a0fff0/demo_custom_dataset.ipynb) [`Download Python source code: demo_custom_dataset.py`](https://deepinv.org/auto_examples/basics/_downloads/a8bfeb8c4310b99b57b67e509d5fb411/demo_custom_dataset.py) [`Download zipped: demo_custom_dataset.zip`](https://deepinv.org/auto_examples/basics/_downloads/1a2c3696b83b1897abce6bce7260e1bd/demo_custom_dataset.zip) [Gallery generated by Sphinx-Gallery](https://sphinx-gallery.github.io)