A Theory of Generative ConvNet

Jianwen Xie* , Yang Lu* , Song-Chun Zhu , and Ying Nian Wu
* Equal contributions.

University of California, Los Angeles (UCLA), USA

Introduction

We show that a generative random field model, which we call generative ConvNet, can be derived from the commonly used discriminative ConvNet, by assuming a ConvNet for multi-category classification and assuming one of the categories is a base category generated by a reference distribution. If we further assume that the non-linearity in the ConvNet is Rectified Linear Unit (ReLU) and the reference distribution is Gaussian white noise, then we obtain a generative ConvNet model that is unique among energy-based models: The model is piecewise Gaussian, and the means of the Gaussian pieces are defined by an auto-encoder, where the filters in the bottom-up encoding become the basis functions in the top-down decoding, and the binary activation variables detected by the filters in the bottom-up convolution process become the coefficients of the basis functions in the top-down deconvolution process. The Langevin dynamics for sampling the generative ConvNet is driven by the reconstruction error of this auto-encoder. The contrastive divergence learning of the generative ConvNet reconstructs the training images by the auto-encoder. The maximum likelihood learning algorithm can synthesize realistic natural image patterns.

Paper

Paper can be downloaded from here.

Code and Data

The code and data can be downloaded from here. For more details, please refer to the instruction.

Experiment 1: Generating Texture Patterns

Figure 1. Generating texture patterns. For each category, the first image displays one of the training images, and the second displays one of the generated images. Please click on the category names or images for more details.

Experiment 2:Generateing Object Patterns

Figure 2. Generating object patterns. For each category, the first image is the training image, and the second image is generated images. Please click on the category names or images for more details.

Experiment 3: Reconstruction by Constrast Divergence Learning

Figure 3. Reconstruction images. For each category, the first image is the training image, and the second image is reconstruction image. Please click on the category names or images for more details.

Acknowledgement

The code in our work is based on the Matlab code of MatConvNet, We thank the authors for sharing their code with the community.

We thank Jifeng Dai for earlier collaboration on generative ConvNet. WWe thank Wenze Hu for earlier collaboration on non-stationary FRAME model. The work is supported by NSF DMS 1310391, ONR MURI N00014-10-1-0933 and DARPA SIMPLEX N66001-15-C-4035.

Top