Jianwen Xie ^{1*},
Zilong Zheng ^{2*},
Ruiqi Gao ^{2},
Wenguan Wang ^{2,3},
Song-Chun Zhu ^{2},
and Ying Nian Wu ^{2}

(* Equal contributions)

^{1} Cognitive Computing Lab, Baidu Research, USA

^{2} University of California, Los Angeles (UCLA), USA

^{3} ETH Zurich, Switzerland

3D data that contains rich geometry information of objects and scenes is valuable for understanding 3D physical world. With the recent emergence of large-scale 3D datasets, it becomes increasingly crucial to have a powerful 3D generative model for 3D shape synthesis and analysis. This paper proposes a deep 3D energy-based model to represent volumetric shapes. The maximum likelihood training of the model follows an “analysis by synthesis” scheme. The benefits of the proposed model are six-fold: first, unlike GANs and VAEs, the model training does not rely on any auxiliary models; second, the model can synthesize realistic 3D shapes by Markov chain Monte Carlo (MCMC); third, the conditional model can be applied to 3D object recovery and super-resolution; fourth, the model can serve as a building block in a multi-grid modeling and sampling framework for high resolution 3D shape synthesis; fifth, the model can be used to train a 3D generator via MCMC teaching; sixth, the unsupervisedly trained model provides a powerful feature extractor for 3D data, which is useful for 3D object classification. Experiments demonstrate that the proposed model can generate high-quality 3D shape patterns and can be useful for a wide variety of 3D shape analysis.

The TPAMI journal paper can be downloaded here.

The TAPMI tex file can be downloaded here.

The CVPR conference paper can be downloaded here.

The CVPR tex file can be downloaded here.

The poster can be downloaded here.

The Python code using tensorflow can be downloaded here

If you wish to use our code, please cite the following paper:

Jianwen Xie*, Zilong Zheng*, Ruiqi Gao, Wenguan Wang, Song-Chun Zhu, Ying Nian Wu

Jianwen Xie*, Zilong Zheng*, Ruiqi Gao, Wenguan Wang, Song-Chun Zhu, Ying Nian Wu

Contents

We evaluate a 3D generator trained by a 3D EBM via cooperative learning scheme on experiments of latent space interpolation and 3D object arithmetic.

Exp 4.1: Interpolation

Exp 4.2: 3D Object Arithmetic

The following shows 3D object arithmetic by the 3D generator net. It encodes semantic knowledge of 3D shapes in the latent space.

We first train a single model on all categories of the training set of ModelNet10 dataset in an unsupervised manner. Then we use the model as a feature extractor. We train a multinomial logistic regression classifier from labeled data based on the extracted feature vectors for classification. The following shows 3D object classification results on ModelNet10 dataset. We evaluate the classification accuracy on the testing data using the one-versus-all rule.

[1] Jianwen Xie*, Yang Lu*, Song-Chun Zhu, Ying Nian Wu. "A Theory of Generative ConvNet." *International Conference on Machine Learning*. 2016. (*equal contribution)

[2] Jianwen Xie, Yang Lu, Ruiqi Gao, Song-Chun Zhu, Ying Nian Wu. "Cooperative Training of Descriptor and Generator Networks." *IEEE Transactions on Pattern Analysis and Machine Intelligence*. 2018.

[3] Jianwen Xie, Yang Lu, Ruiqi Gao, Song-Chun Zhu, Ying Nian Wu. "Cooperative Learning of Energy-Based Model and Latent Variable Model via MCMC Teaching." *The Thirty-Second AAAI Conference on Artificial Intelligence*. 2018.

[4] Ruiqi Gao, Ruiqi Gao, Yang Lu, Junpei Zhou, Song-Chun Zhu, Ying Nian Wu. "Learning Generative ConvNets via Multi-Grid Modeling and Sampling." *Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition*. 2018.