Generative PointNet: Deep Energy-Based Learning on Unordered Point Sets for 3D Generation, Reconstruction and Classification

Jianwen Xie 1*, Yifei Xu2*, Zilong Zheng 2, Song-Chun Zhu 2,3,4, and Ying Nian Wu 2

(* Equal contributions)
1 Cognitive Computing Lab, Baidu Research, Bellevue, USA
2 University of California, Los Angeles (UCLA), USA
3 Tsinghua University, Beijing, China
4 Peking University, Beijing, China


Abstract

We propose a generative model of unordered point sets, such as point clouds, in the form of an energy-based model, where the energy function is parameterized by an input-permutation-invariant bottom-up neural network. The energy function learns a coordinate encoding of each point and then aggregates all individual point features into an energy for the whole point cloud. We call our model the generative PointNet because it can be derived from the discriminative PointNet. Our model can be trained by MCMC-based maximum likelihood learning (as well as its variants), without the help of any assisting networks like those in GANs and VAEs. Unlike most point cloud generators that rely on hand-crafted distance metrics, our model does not require any hand-crafted distance metric for the point cloud generation, because it synthesizes point clouds by matching observed examples in terms of statistical properties defined by the energy function. Furthermore, we can learn a short-run MCMC toward the energy-based model as a flow-like generator for point cloud reconstruction and interpolation. The learned point cloud representation can be useful for point cloud classification. Experiments demonstrate the advantages of the proposed generative model of point clouds.

Energy-Based Generative PointNet

Paper

The CVPR conference paper can be downloaded here.

The CVPR tex file can be downloaded here.

Code and Data

The Python code using pytorch can be downloaded here

We use ModelNet10 as our training data. You can download our pre-processed dataset here

The full pretrained checkpoint is available here. You can first download one category chair: here

The checkpoint we used for classification can be downloaded here.

If you wish to use our code, please cite the following paper: 

Generative PointNet: Deep Energy-Based Learning on Unordered Point Sets for 3D Generation, Reconstruction and Classification
Jianwen Xie*, Yifei Xu*, Zilong Zheng, Song-Chun Zhu, Ying Nian Wu
IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 2021

Reference

Deep Energy-Based Models for images, videos, and voxels can be found as follows:.

[1] Jianwen Xie*, Yang Lu*, Song-Chun Zhu, Ying Nian Wu. "A Theory of Generative ConvNet." International Conference on Machine Learning (ICML). 2016. (*equal contribution)

[2] Jianwen Xie, Song-Chun Zhu, Ying Nian Wu. "Synthesizing Dynamic Pattern by Spatial-Temporal Generative ConvNet." IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 2017.

[3] Jianwen Xie*, Zilong Zheng*, Ruiqi Gao, Wenguan Wang, Song-Chun Zhu, Ying Nian Wu. "Learning Descriptor Networks for 3D Shape Synthesis and Analysis." IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 2018.

[4] Jianwen Xie, Song-Chun Zhu, Ying Nian Wu. "Learning Energy-based Spatial-Temporal Generative ConvNet for Dynamic Patterns." IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI). 2019.

[5] Jianwen Xie*, Zilong Zheng*, Ruiqi Gao, Wenguan Wang, Song-Chun Zhu, Ying Nian Wu. "Generative VoxelNet: Learning Energy-Based Models for 3D Shape Synthesis and Analysis." IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 2020.

Acknowledgment

We thank Erik Nijkamp for insightful discussions about short-run MCMC for EBM and neural tangent kernel.

Top