#### NOTE
New to DeepInverse? Get started with the basics with the
[5 minute quickstart tutorial](https://deepinv.org/auto_examples/basics/demo_quickstart.html.md#sphx-glr-auto-examples-basics-demo-quickstart-py)..
# Remote sensing with satellite images
In this example we demonstrate remote sensing inverse problems for multispectral satellite imaging.
These have important applications for image restoration in environmental monitoring, urban planning, disaster recovery etc.
We will demonstrate pan-sharpening, i.e., recovering high-resolution multispectral images from measurement pairs of
low-resolution multispectral images and high-resolution panchromatic (single-band) images with the forward
operator [`deepinv.physics.Pansharpen`](https://deepinv.org/api/stubs/deepinv.physics.Pansharpen.html.md#deepinv.physics.Pansharpen).
We will also demonstrate other inverse problems including compressive spectral imaging and hyperspectral unmixing.
We provide a convenient satellite image dataset for pan-sharpening [`deepinv.datasets.NBUDataset`](https://deepinv.org/api/stubs/deepinv.datasets.NBUDataset.html.md#deepinv.datasets.NBUDataset) provided in the paper Meng *et al.*[1](#footcite-meng2020large).
which includes data from several satellites such as WorldView satellites.
#### TIP
For remote sensing experiments, DeepInverse provides the following classes:
- [`Pan-sharpening`](https://deepinv.org/api/stubs/deepinv.physics.Pansharpen.html.md#deepinv.physics.Pansharpen)
- [`Compressive spectral imaging`](https://deepinv.org/api/stubs/deepinv.physics.CompressiveSpectralImaging.html.md#deepinv.physics.CompressiveSpectralImaging)
- [`Hyperspectral unmixing`](https://deepinv.org/api/stubs/deepinv.physics.HyperSpectralUnmixing.html.md#deepinv.physics.HyperSpectralUnmixing)
- [`Super resolution`](https://deepinv.org/api/stubs/deepinv.physics.Downsampling.html.md#deepinv.physics.Downsampling)
- [`Satellite imagery dataset`](https://deepinv.org/api/stubs/deepinv.datasets.NBUDataset.html.md#deepinv.datasets.NBUDataset)
- Metrics for multispectral data: [`QNR`](https://deepinv.org/api/stubs/deepinv.loss.metric.QNR.html.md#deepinv.loss.metric.QNR), [`SpectralAngleMapper`](https://deepinv.org/api/stubs/deepinv.loss.metric.SpectralAngleMapper.html.md#deepinv.loss.metric.SpectralAngleMapper), [`ERGAS`](https://deepinv.org/api/stubs/deepinv.loss.metric.ERGAS.html.md#deepinv.loss.metric.ERGAS)
```Python
import deepinv as dinv
import torch
device = dinv.utils.get_device()
```
```none
Selected GPU 0 with 17332.4375 MiB free memory
```
## Load raw pan-sharpening measurements
The dataset includes raw pansharpening measurements
containing `(MS, PAN)` where `MS` are the low-res (4-band) multispectral and `PAN` are the high-res
panchromatic images. Note there are no ground truth images!
#### NOTE
The pan-sharpening measurements are provided as a [`deepinv.utils.TensorList`](https://deepinv.org/api/stubs/deepinv.utils.TensorList.html.md#deepinv.utils.TensorList), since
the pan-sharpening physics [`deepinv.physics.Pansharpen`](https://deepinv.org/api/stubs/deepinv.physics.Pansharpen.html.md#deepinv.physics.Pansharpen) is a stacked physics combining
[`deepinv.physics.Downsampling`](https://deepinv.org/api/stubs/deepinv.physics.Downsampling.html.md#deepinv.physics.Downsampling) and [`deepinv.physics.Decolorize`](https://deepinv.org/api/stubs/deepinv.physics.Decolorize.html.md#deepinv.physics.Decolorize).
See the User Guide [Combining Physics](https://deepinv.org/user_guide/physics/intro.html.md#physics-combining) for more information.
Note, for plotting purposes we only plot the first 3 bands (RGB).
Note also that the linear adjoint must assume the unknown spectral response function (SRF).
```Python
DATA_DIR = dinv.utils.get_cache_home() / "demo_remote_sensing"
dataset = dinv.datasets.NBUDataset(DATA_DIR, return_pan=True, download=True)
y = dataset[0].unsqueeze(0).to(device) # MS (1,4,256,256), PAN (1,1,1024,1024)
physics = dinv.physics.Pansharpen((4, 1024, 1024), factor=4, device=device)
# Pansharpen with classical Brovey method
x_hat = physics.A_dagger(y) # shape (1,4,1024,1024)
dinv.utils.plot(
[
y[0][:, :3],
y[1], # Note this will be interpolated to match high-res image size
x_hat[:, :3],
physics.A_adjoint(y)[:, :3],
],
titles=[
"Input MS",
"Input PAN",
"Pseudo-inverse \n using \n Brovey method",
"Linear adjoint",
],
dpi=1200,
)
```
```none
/local/pberna03/actions-runner/_work/deepinv/deepinv/deepinv/datasets/satellite.py:106: DeprecationWarning: The tuple format for dataset outputs is deprecated and will be removed in a future version.It is recommended to set `use_dict_output=True` for better readability and flexibility in returned outputs.The default is currently `False` for backward compatibility, but will be switched to `True` in a future version.
super().__init__(use_dict_output=use_dict_output)
Downloading /local/pberna03/.cache/deepinv/demo_remote_sensing/gaofen-1.zip
0%| | 0/7914941 [00:00
Evaluate performance - note we can only use QNR as we have no GT
```Python
qnr = dinv.metric.QNR()
print(qnr(x_net=x_hat, x=None, y=y, physics=physics))
```
```none
tensor([0.3534], device='cuda:0')
```
## Simulate remote-sensing measurements
We can also simulate measurements from various remote sensing inverse problems so that we have pairs of
measurements and ground truth. Now, the dataset loads ground truth images `x`:
```Python
dataset = dinv.datasets.NBUDataset(DATA_DIR, return_pan=False)
x = dataset[0].unsqueeze(0).to(device) # just MS of shape 1,4,256,256
```
```none
/local/pberna03/actions-runner/_work/deepinv/deepinv/deepinv/datasets/satellite.py:106: DeprecationWarning: The tuple format for dataset outputs is deprecated and will be removed in a future version.It is recommended to set `use_dict_output=True` for better readability and flexibility in returned outputs.The default is currently `False` for backward compatibility, but will be switched to `True` in a future version.
super().__init__(use_dict_output=use_dict_output)
```
For **compressive spectral imaging**, we use the coded-aperture snapshot spectral imaging (CASSI) model,
which is a popular hyperspectral imaging method. See [`deepinv.physics.CompressiveSpectralImaging`](https://deepinv.org/api/stubs/deepinv.physics.CompressiveSpectralImaging.html.md#deepinv.physics.CompressiveSpectralImaging)
```Python
physics = dinv.physics.CompressiveSpectralImaging(x.shape[1:], mode="sd", device=device)
y = physics(x) # 1,1,256,256
dinv.utils.plot([x[:, :3], y], titles=["Image x", "CASSI meas. y"])
```
For **hyperspectral unmixing**, our images are the measurements and we seek to recover abundances
given the endmember matrix in the linear mixing model.
In this toy example, we perform unmixing with 2 endmembers: one purely yellow and one purely blue.
```Python
physics = dinv.physics.HyperSpectralUnmixing(
M=torch.tensor([[0.5, 0.5, 0.0, 0.0], [0.0, 0.0, 1.0, 0.0]]), device=device
)
abundance = physics.A_adjoint(x) # 1,2,256,256
dinv.utils.plot(
[x[:, :3], abundance[:, [0]], abundance[:, [1]]],
titles=["Mixed image", "Yellow abundance", "Blue abundance"],
)
```
For the **pansharpening** physics, we assume a flat spectral response function,
but this can also be jointly learned. We simulate Gaussian noise on the panchromatic images.
```Python
physics = dinv.physics.Pansharpen((4, 256, 256), factor=4, srf="flat", device=device)
y = physics(x)
# Pansharpen with classical Brovey method
x_hat = physics.A_dagger(y)
```
## Solving pan-sharpening with neural networks
The pan-sharpening physics is compatible with the rest of the DeepInverse library
so we can solve the inverse problem using any method provided in the library.
For example, we use here the PanNet Yang *et al.*[2](#footcite-yang2017pannet) model.
This model can be trained using losses such as supervised learning using [`deepinv.loss.SupLoss`](https://deepinv.org/api/stubs/deepinv.loss.SupLoss.html.md#deepinv.loss.SupLoss)
or self-supervised learning using Equivariant Imaging [`deepinv.loss.EILoss`](https://deepinv.org/api/stubs/deepinv.loss.EILoss.html.md#deepinv.loss.EILoss), which was applied to
pan-sharpening in Wang and Davies[3](#footcite-wang2024perspective).
For evaluation, we use the standard full-reference metrics (ERGAS, SAM) and no-reference (QNR).
#### NOTE
This is a tiny example using 5 images. We demonstrate training for 1 epoch for speed, but you can train from scratch using 50 epochs.
```Python
model = dinv.models.PanNet(hrms_shape=(4, 256, 256), device=device)
x_net = model(y, physics)
```
Example training loss using measurement consistency on the multispectral images
and Stein’s Unbiased Risk Estimate on the panchromatic images.
For metrics, we use standard full-reference and no-reference multispectral pan-sharpening metrics,
since ground-truth is now available.
```Python
loss = dinv.loss.StackedPhysicsLoss(
[dinv.loss.MCLoss(), dinv.loss.SureGaussianLoss(0.05)]
)
sam = dinv.metric.distortion.SpectralAngleMapper()
ergas = dinv.metric.distortion.ERGAS(factor=4)
qnr = dinv.metric.QNR()
print(sam(x_hat, x), ergas(x_hat, x), qnr(x_hat, x=None, y=y, physics=physics))
```
```none
tensor([0.0543], device='cuda:0') tensor([1.7147], device='cuda:0') tensor([0.5415], device='cuda:0')
```
For training, we first load optimizer and pretrained model,
then train using the deepinv Trainer.
```Python
optimizer = torch.optim.Adam(model.parameters())
from deepinv.models.utils import get_weights_url
file_name = "demo_nbu_pansharpen.pth"
url = get_weights_url(model_name="demo", file_name=file_name)
ckpt = torch.hub.load_state_dict_from_url(
url, map_location=lambda storage, loc: storage, file_name=file_name
)
model.load_state_dict(ckpt["state_dict"])
optimizer.load_state_dict(ckpt["optimizer"])
from torch.utils.data import DataLoader
trainer = dinv.Trainer(
model=model,
physics=physics,
optimizer=optimizer,
losses=loss,
metrics=[sam, ergas],
train_dataloader=DataLoader(dataset),
epochs=1,
online_measurements=True,
plot_images=False,
compare_no_learning=True,
no_learning_method="A_dagger",
show_progress_bar=False,
device=device,
)
trainer.train()
trainer.test(DataLoader(dataset))
```
```none
Downloading: "https://huggingface.co/deepinv/demo/resolve/main/demo_nbu_pansharpen.pth?download=true" to /local/pberna03/.cache/torch/hub/checkpoints/demo_nbu_pansharpen.pth
0%| | 0.00/954k [00:00
Plot sample results:
```Python
dinv.utils.plot(
[
x[:, :3],
y[0][:, :3],
y[1],
x_hat[:, :3],
x_net[:, :3],
],
titles=["x HRMS", "y LRMS", "y PAN", "Estimate (classical)", "Estimate (PanNet)"],
)
```
* **References:**
* Xiangchao Meng, Yiming Xiong, Feng Shao, Huanfeng Shen, Weiwei Sun, Gang Yang, Qiangqiang Yuan, Randi Fu, and Hongyan Zhang. A large-scale benchmark data set for evaluating pansharpening performance: overview and implementation. *IEEE Geoscience and Remote Sensing Magazine*, 9(1):18–52, 2020.
* Junfeng Yang, Xueyang Fu, Yuwen Hu, Yue Huang, Xinghao Ding, and John Paisley. Pannet: a deep network architecture for pan-sharpening. In *Proceedings of the IEEE international conference on computer vision*, 5449–5457. 2017.
* Andrew Wang and Mike Davies. Perspective-equivariant imaging: an unsupervised framework for multispectral pansharpening. *arXiv e-prints*, pages arXiv–2403, 2024.
**Total running time of the script:** (0 minutes 8.518 seconds)
🏷 Tags: Remote sensing, Super-resolution
[`Download Jupyter notebook: demo_remote_sensing.ipynb`](https://deepinv.org/auto_examples/physics/_downloads/776c9c6276194c97f380ed49ba9537c9/demo_remote_sensing.ipynb)
[`Download Python source code: demo_remote_sensing.py`](https://deepinv.org/auto_examples/physics/_downloads/282f97bed3303f0030597f70cbd1bd5d/demo_remote_sensing.py)
[`Download zipped: demo_remote_sensing.zip`](https://deepinv.org/auto_examples/physics/_downloads/14ef1a71a81f10ecc50650cb0947a7c1/demo_remote_sensing.zip)
[Gallery generated by Sphinx-Gallery](https://sphinx-gallery.github.io)