Nvidia's GauGAN technology instantly turns any drawing into a work of art

The name of the new image processing technology "GauGAN" contains a reference to the impressionist artist Gauguin and to GAN (generative adversarial networks). According to the CEO of nVidia, Jensen Huang, it is the GAN that will solve the key problem in creating artificial intelligence: how to effectively build the learning process for such a system? GauGAN technology serves to demonstrate this approach.

All GANs have a narrow specialization - and in the case of GauGAN, this is drawing photorealistic landscapes. A schematic sketch is taken as a basis, where a blue line can mean a river or canal, a white spot on top of a cloud or a balloon, yellow lines - fallen leaves or a wheat field, etc. The system analyzes the composition of the painting and selects fragments from real photographs and drawings, achieving their ideal combination with each other.

GauGAN knows how to take into account many important details, such as the location of shadows from objects, waves on the water, the size of buildings in relation to landscape elements. If a large white spot is identified as a snowy clearing, the sky will be dark leaden or heavy clouds, as it should be in winter. This is very important, because the GauGAN example also works out the feedback - neural networks learn to recognize the general image, starting from the position in the frame of an already known object.

Such feedback will help, for example, to teach the autopilot to determine the presence of precipitation on the road if pedestrians take out umbrellas. And in video games, technology will help to draw on the fly what the player expects to see in a certain situation, adding realism or outlandish special effects. It is possible that the next step will be the creation of entire 3D worlds, and then the turn of the video will come. And then, experts fear, artificial intelligence will finally learn to blur the line between fiction and reality.