# Flow Contrastive Estimation of Energy-Based Models

Ruiqi Gao ^{1},
Erik Nijkamp ^{1},
Diederik P. Kingma ^{2},
Zhen Xu ^{2},
Andrew M. Dai ^{2},
Ying Nian Wu ^{1}

^{1} University of California, Los Angeles (UCLA)

^{2} Google Brain

## Abstract

This paper studies a training method to jointly estimate an energy-based model and a flow-based model, in which the two models are iteratively updated based on a shared adversarial value function. This joint training method has the following traits. (1) The update of the energy-based model is based on noise contrastive estimation, with the flow model serving as a strong noise distribution. (2) The update of the flow model approximately minimizes the Jensen-Shannon divergence between the flow model and the data distribution. (3) Unlike generative adversarial networks (GAN) which estimates an implicit probability distribution defined by a generator model, our method estimates two explicit probabilistic distributions on the data. Using the proposed method we demonstrate a significant improvement on the synthesis quality of the flow model, and show the effectiveness of unsupervised feature learning by the learned energy-based model. Furthermore, the proposed training method can be easily adapted to semi-supervised learning. We achieve competitive results to the state-of-the-art semi-supervised learning methods.

## Paper

The paper can be downloaded here.

## Code

The TensorFlow2 code is coming soon!

If you wish to use the code or results, please cite the following paper:

**Flow Contrastive Estimation of Energy-Based Models**

@article{gao2019flow,

title={Flow Contrastive Estimation of Energy-Based Models},

author={Gao, Ruiqi and Nijkamp, Erik and Kingma, Diederik P and Xu, Zhen and Dai, Andrew M and Wu, Ying Nian},

journal={arXiv preprint arXiv:1912.00589},

year={2019}}

## Experiments

**Exp 1 **:

Density estimation on 2D synthetic data
**Exp 2 **:

Learning on real image datasets
**Exp 3 **:

Unsupervised feature learning
**Exp 4 **:

Semi-supervised learning

### Experiment 1: Density estimation on 2D synthetic data

### Experiment 2: Learning on real image datasets

### Experiment 3: Unsupervised feature learning

### Experiment 4: Semi-supervised learning

### Acknowledgment

The work is partially supported by DARPA XAI project N66001-17-2-4029 and ARO project W911NF1810296. We thank Pavel Sountsov, Alex Alemi, Matthew D. Hoffman and Srinivas Vasudevan for their helpful discussions.