Label-Conditional Synthetic Satellite Imagery

Label conditional synthetic satellite imagery generation

We adapt conditional GAN with SPADE to the synthetic satellite imagery task and optimize it to generate better images, prepare different datasets for training the downstream models, and evaluate the performances of the downstream segmentation models trained on different datasets.

We trained SPADE using real 3-channel VHR satellite imagery, land cover and building segmentation masks as inputs. Since the baseline model tends to produce almost identical synthetic images given the same masks, we include in training an additional loss term to increase output diversity.

Our results suggest that increasing diversity leads to more photo-realistic synthetic images and better downstream performance in segmentation. Synthetic satellite images are generated at different levels of diversity for our downstream experiments (please refer to our paper for more details).

Above, each row shows three examples of synthetic images generated from random latent representations for a given class mask from the model. The “real imagery” is shown for reference but is not used at inference time.

Downstream Tasks

To test whether our synthetic satellite imagery could be effective data augmentation, we design a series of experiments based on the multi-label landcover segmentation task, where a downstream model classifies an image’s pixels into 6 different land cover labels.

We first explore the segmentation performance of models trained with synthetic imagery with different degrees of diversity, as generated with different lambda diversities.

mIoU reflects the downstream model performance trained on generated images with diversity lambda. FID (1) is calculated with synthetic test images generated without using the trained encoder. FID (2) is calculated with synthetic test images generated using the trained encoder.

Synthetic tiles generated with lambda = 6 results in the highest mIoU scores of 0.4059. We use this diversity parameter value to generate synthetic tiles for most of the following experiments for consistently better performance.

To evaluate the usability of the synthetic images with respect to the real ones, we trained four downstream segmentation models on the following datasets: 100% real, which contains 100 real satellite image tiles in a 3-channel RGB version; 100% syn, 200% syn, 300% syn, which contains 100, 200, 300 synthetic RGB tiles respectively (we generate 3 different synthetic versions for each real tile by changing the input random setting during upstream generation). All synthetic tiles are generated with the upstream model using lambda=6, and the downstream model randomly crops out patches from real and synthetic tiles for training.

The mIoU metric below shows that training on only and more synthetic imagery does not improve segmentation performance, and specifically, the performance on the water class drops significantly.

Since the NIR channel contains a lot of information about water bodies and thus including it might improve the segmentation performance, we also trained two models on 100% real (4-channel) and 100% synthetic (4-channel). The results are also shown above, please refer to our paper for more details.

Even with only 10% amount of data, the segmentation model trained on 4-channel synthetic images could reach comparable perfor- mance as the model trained on 3-channel real images.

Furthermore, we combine real and synthetic images using different mix-proportions to explore empirically whether or not and how much inclusion of synthetic satellite imagery is a better augmentation strategy in training downstream segmentation models. As shown by the figure below, the model trained with the dataset containing 50% synthetic images could reach higher mIoU (0.5834) and hence better segmentation performance, compared to the model trained only with real images (0.5235).