In&Out: Diverse Image Outpainting via GAN Inversion


Outpaint
Real Image
Outpaint

Abstract
Image outpainting seeks for a semantically consistent extension of the input image beyond its available content. Compared to inpainting --- filling in missing pixels in a way coherent with the neighboring pixels --- outpainting can be achieved in more diverse ways since the problem is less constrained by the surrounding pixels. Existing image outpainting methods pose the problem as a conditional image-to-image translation task, often generating repetitive structures and textures by replicating the content available in the input image. In this work, we formulate the problem from the perspective of inverting generative adversarial networks. Our generator renders micro-patches conditioned on their joint latent code as well as their individual positions in the image. To outpaint an image, we seek for multiple latent codes not only recovering available patches but also synthesizing diverse outpainting by patch-based generation. This leads to richer structure and content in the outpainted regions. Furthermore, our formulation allows for outpainting conditioned on the categorical input, thereby enabling flexible user controls. Extensive experimental results demonstrate the proposed method performs favorably against existing in- and outpainting methods, featuring higher visual quality and diversity.

Paper

Citation

Yen-Chi Cheng, Chieh Hubert Lin, Hsin-Ying Lee, Jian Ren, Sergey Tulyakov, Ming-Hsuan Yang, "In&Out: Diverse Image Outpainting via GAN Inversion", in arXiv, 2021.


Bibtex
@article{InOut,
    author    = {Cheng, Yen-Chi and Lin, Chieh Hubert and Lee, Hsin-Ying and Ren, Jian and Tulyakov, Sergey and Yang, Ming-Hsuan}, 
    title     = {In&Out: Diverse Image Outpainting via GAN Inversion},
    volume    = {abs/2104.00675},
    year      = {2021}
}
Overview
Results
Flickr-Landscape
Outpainting results of the collected Flickr-Landscape dataset.
LSUN-Church
Outpainting results of the church cateogry of the LSUN dataset.

Comparisons
The proposed approach is more stable, synthesizes richer context with more complex structures, and is able to handle some of the difficultcomplex scenes.

Multi-directional and irregular-boundary outpainting
The proposed method can naturally handle outpainting to any direction and with irregular boundaries.

Panorama
The proposed method can perform panorama generation of high quality and high structural complexity without repeating patterns.
References