University of California, Los Angeles (UCLA), USA
We show that a generative random field model, which we call generative ConvNet, can be derived from the commonly used discriminative ConvNet, by assuming a ConvNet for multi-category classification and assuming one of the categories is a base category generated by a reference distribution. If we further assume that the non-linearity in the ConvNet is Rectified Linear Unit (ReLU) and the reference distribution is Gaussian white noise, then we obtain a generative ConvNet model that is unique among energy-based models: The model is piecewise Gaussian, and the means of the Gaussian pieces are defined by an auto-encoder, where the filters in the bottom-up encoding become the basis functions in the top-down decoding, and the binary activation variables detected by the filters in the bottom-up convolution process become the coefficients of the basis functions in the top-down deconvolution process. The Langevin dynamics for sampling the generative ConvNet is driven by the reconstruction error of this auto-encoder. The contrastive divergence learning of the generative ConvNet reconstructs the training images by the auto-encoder. The maximum likelihood learning algorithm can synthesize realistic natural image patterns.
Paper can be downloaded from here.
Figure 1. Generating texture patterns. For each category, the first image displays one of the training images, and the second displays one of the generated images. Please click on the category names or images for more details.
Figure 2. Generating object patterns. For each category, the first image is the training image, and the second image is generated images. Please click on the category names or images for more details.
Figure 3. Reconstruction images. For each category, the first image is the training image, and the second image is reconstruction image. Please click on the category names or images for more details.
The code in our work is based on the Matlab code of MatConvNet, We thank the authors for sharing their code with the community.
We thank Jifeng Dai for earlier collaboration on generative ConvNet. WWe thank Wenze Hu for earlier collaboration on non-stationary FRAME model. The work is supported by NSF DMS 1310391, ONR MURI N00014-10-1-0933 and DARPA SIMPLEX N66001-15-C-4035.