(* Equal contributions)
1 Cognitive Computing Lab, Baidu Research, USA
2 University of California, Los Angeles (UCLA), USA
3 Massachusetts Institute of Technology, USA
This paper studies the problem of learning the conditional distribution of a high-dimensional output given an input,e.g., the output is a photo image and the input is a sketch image. We solve this problem by cooperative training of a fast thinking initializer and slow thinking solver. The initializer generates the output directly by a non-linear transformation of the input as well as a noise vector that accounts for latent variability in the output. The slow thinking solver learns an objective function in the form of a conditional energy function, so that the output can be generated by optimizing the objective function, or more rigorously by sampling from the conditional energy-based model. We propose to learn the two models jointly, where the fast thinking initializer serves to initialize the sampling of the slow thinking solver, and the solver refines the initial output by an iterative algorithm. The solver learns from the difference between the refined output and the observed output, while the initializer learns from how the solver refines its initial output. We demonstrate the effectiveness of the proposed method on various conditional learning tasks, e.g., class-to-image generation, image-to-image translation, and image recovery. The advantage of our method over GAN-based methods is that our method is equipped with a slow thinking process that refines the solution guided by a learned objective function.
The TPAMI journal paper can be downloaded here.
The TAPMI tex file can be downloaded here.
The Python code using tensorflow will be available soon.
If you wish to use our code, please cite the following paper:
ContentsExp 1 : Category => Image
 Jianwen Xie, Yang Lu, Ruiqi Gao, Song-Chun Zhu, Ying Nian Wu. "Cooperative Training of Descriptor and Generator Networks." IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI). 2018.
 Jianwen Xie, Yang Lu, Ruiqi Gao, Song-Chun Zhu, Ying Nian Wu. "Cooperative Learning of Energy-Based Model and Latent Variable Model via MCMC Teaching." The Thirty-Second AAAI Conference on Artificial Intelligence (AAAI). 2018.
 Jianwen Xie*, Yang Lu*, Song-Chun Zhu, Ying Nian Wu. "A Theory of Generative ConvNet." International Conference on Machine Learning (ICML). 2016. (*equal contribution)
 Tian Han*, Yang Lu*, Song-Chun Zhu, Ying Nian Wu. "Alternating Back-Propagation for Generator Network" The Thirty-First AAAI Conference on Artificial Intelligence (AAAI). 2017. (*equal contribution)