Generative adversarial networks (GANs) are deep learning models that are used to generate images similar to real images. Images generated by GANs are both realistic and personalized. But to generate images of high quality, the network requires a huge amount of data. Their usability limits in case of a low quantity of data. In this blog, we will discuss how we can use simulated data to generate images of climate change using GANs in case we are having a scarcity of training data.
Introduction
Recently researchers with the Montreal Institute for Learning Algorithms (MILA) used generative adversarial networks to generate images of the world after the flood. They tried to show how the world would change if some calamity like a flood occurs. They hope people would work to avert future weather conditions if they can see these changes. Researchers used simulated data in combination with real images to train multimodel unsupervised image-to-image translation with some modification to architecture.
Data Collection
Real Dataset
Researchers have collected 2000 real images of flooded and non-flooded scenes taken in various weather conditions, seasons, time and viewpoints. These images were taken from publicly available datasets Mapilary and Flickr. They used this dataset and trained CycleGAN but the generated images were not sufficiently realistic. To cop up with this problem they used simulated data.
Simulated Dataset
To generate simulated dataset researchers used the Unity 3D game engine. They created different types of building in combination with urban and rural environments. As a starting point, they generated 1000 unique pairs of images with flooded and non-flooded domains.
Domain Adaption Technique
While using simulated data, authors have seen domain gap between training dataset made up of simulated data and testing data made up of real images. To bridge this gap they used domain adaption technique inspired by unsupervised semantic segmentation. This technique is being implemented by using an adversarial classifier within MUNIT architecture.
Network Architecture
Researchers have tried different image-to-image translation GANs like CycleGAN, InstaGAN, and MUNIT. CycleGAN and InstaGAN were not able to generate as realistic water texture as MUNIT was able to. Finally, they used MUNIT architecture with some modifications.
MUNIT architecture relies on two generators and two discriminators to disentangle the style and content of the images. Such that during the generation of the image only style changes and content remains the same. To make MUNIT architecture more compatible with climate change use case, researchers have made the following changes to the architecture:
- Restriction of Cycle COnsistency Loss: In image-to-image translation GANs, cycle consistency loss is used to make sure that translation is cycle consistent. Let say, If we translate from English to French and then translate back to English sentence, we should arrive at the original sentence. In this architecture, researchers have restricted the network’s cycle consistency loss such that this loss is only computed on those regions that are not likely to be flooded. To do this they have used the binary masks of the areas.
- Introduction of semantic consistency loss: This loss confirms that the semantic segmentation structure for the generated image is the same as the source image except for the areas where changes occurred like the road to the flooded area.
This approach uses both real and simulated data to perform image-to-image translation to show the effects of climate change. This approach clearly shows that simulated data helps in generating more realistic images. Researchers are still working on to improve the results of this model. They are also working to create an interactive website.
“Authors aim to develop an interactive website that, given a user-entered address, will query the Google Street View API (Anguelov et al., 2010) to get an image of the location and alter it to display a plausible image of its climate future based on the predictions of climate models. We hope this tool will help communicate effectively on climate change related risks.“
Referenced Research Paper: Using Simulated data to generate images of climate change
Hope you enjoy reading.
If you have any doubt/suggestion please feel free to ask and I will do my best to help or improve myself. Good-bye until next time.