Jianwen Xie ^{1,2},
Yang Lu ^{1,3},
Ruiqi Gao ^{1},
and Ying Nian Wu ^{1}

^{1} University of California, Los Angeles (UCLA), USA

^{2} Hikvision Research America

^{3}Amazon RSML (Retail System Machine Learning) Group

This paper proposes a cooperative learning algorithm to train both the undirected energy-based model and the directed latent variable model jointly. The learning algorithm interweaves the maximum likelihood algorithms for learning the two models, and each iteration consists of the following two steps: (1) Modified contrastive divergence for energy-based model: The learning of the energy-based model is based on the contrastive divergence, but the finite-step MCMC sampling of the model is initialized from the synthesized examples generated by the latent variable model instead of being initialized from the observed examples. (2) MCMC teaching of the latent variable model: The learning of the latent variable model is based on how the MCMC in (1) changes the initial synthesized examples generated by the latent variable model, where the latent variables that generate the initial synthesized examples are known so that the learning is essentially supervised. Our experiments show that the cooperative learning algorithm can learn realistic models of images.

The paper can be downloaded here.

The oral presentation can be downloaded here.

The code and data for CoopNet can be downloaded here.

The code and data for recovery experiment can be downloaded here.

Contents Exp 1 : Experiment on texture synthesis (homogeneous CoopNets)Exp 2 : Experiment on scene and object synthesis (inhomogeneous CoopNets)Exp 3 : Experiment on pattern completion |

n=50 | n=100 | n=300 | n=500 | n=700 | n=900 | n=1100 | |

CoopNets | 2.66 ± .13 |
3.04 ± .13 |
3.41 ± .13 |
3.48 ± .08 |
3.59 ± .11 |
3.65 ± .07 |
3.79 ± .15 |

DCGAN | 2.26 ± .16 | 2.50 ± .15 | 3.16 ± .15 | 3.05 ± .12 | 3.13 ± .09 | 3.34 ± .05 | 3.47 ± .06 |

EBGAN | 2.23 ± .17 | 2.40 ± .14 | 2.62 ± .08 | 2.46 ± .09 | 2.65 ± .04 | 2.64 ± .04 | 2.75 ± .08 |

W-GAN | 1.80 ± .09 | 2.19 ± .12 | 2.34 ± .06 | 2.62 ± .08 | 2.86 ± .10 | 2.88 ± .07 | 3.14 ± .06 |

VAE | 1.62 ± .09 | 1.63 ± .06 | 1.65 ± .05 | 1.73 ± .04 | 1.67 ± .03 | 1.72 ± .02 | 1.73 ± .02 |

InfoGAN | 2.21 ± .04 | 1.73 ± .01 | 2.15 ± .03 | 2.42 ± .05 | 2.47 ± .05 | 2.29 ± .03 | 2.08 ± .04 |

DDGM | 2.65 ± .17 | 1.05 ± .03 | 3.27 ± .14 | 3.42 ± .09 | 3.47 ± .13 | 3.41 ± .08 | 3.34 ± .11 |

Algorithm G | 1.72 ± .07 | 1.94 ± .09 | 2.32 ± .09 | 2.40 ± .06 | 2.45 ± .05 | 2.54 ± .05 | 2.61 ± .06 |

Persistent CD | 1.30 ± .08 | 1.94 ± .03 | 1.80 ± .02 | 1.53 ± .02 | 1.45 ± .04 | 1.35 ± .02 | 1.51 ± .02 |

Exp | task | CoopNets | DCGAN | MRF-L1 | MRF-L2 | inter-1 | inter-2 | inter-3 | inter-4 | inter-5 |

error | M30 | 0.115 |
0.211 | 0.132 | 0.134 | 0.120 | 0.120 | 0.265 | 0.120 | 0.120 |

M40 | 0.124 |
0.212 | 0.148 | 0.149 | 0.135 | 0.135 | 0.314 | 0.135 | 0.135 | |

M50 | 0.136 |
0.214 | 0.178 | 0.179 | 0.170 | 0.166 | 0.353 | 0.164 | 0.164 | |

PSNR | M30 | 16.893 |
12.116 | 15.739 | 15.692 | 16.203 | 16.635 | 9.524 | 16.665 | 16.648 |

M40 | 16.098 |
11.984 | 14.834 | 14.785 | 15.065 | 15.644 | 8.178 | 15.698 | 15.688 | |

M50 | 15.105 |
11.890 | 13.313 | 13.309 | 13.220 | 14.009 | 7.327 | 14.164 | 14.161 |

We acknowledge Dr. Song-Chun Zhu’s important contributions to the work presented in this paper. We thank a reviewer for his or her insightful comments. We thank Hansheng Jiang for her work on this project as a summer visiting student. We thank Tengyu Liu and Zilong Zheng for assistance with the inception score comparison experiments. The work is supported by NSF DMS 1310391, DARPA SIMPLEX N66001-15-C-4035, ONR MURI N00014-16-1- 2007, and DARPA ARO W911NF-16-1-0579.