(* Equal contributions)
1 Hikvision Research Institute, Santa Clara, USA
2 University of California, Los Angeles (UCLA), USA
3 Beijing Institute of Technology, China
The paper can be downloaded here.
The tex file can be downloaded here.
The poster can be downloaded here.
The oral presentation can be downloaded here.
Code and Data
The Python code using tensorflow is coming soon
If you wish to use our code, please cite the following paper:
Learning Descriptor Networks for 3D Shape Synthesis and Analysis
Jianwen Xie*, Zilong Zheng*, Ruiqi Gao, Wenguan Wang, Song-Chun Zhu, Ying Nian Wu
IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 2018 (Oral)
Experiment 1: Generating 3D Objects
We conduct experiments on synthesizing 3D objects of
categories from ModelNet dataset.
Figure1. Generating 3D objects. For each category, the first three 3D objects are observed examples, columns 4, 5, 6, 7, 8, and 9 are 6 of the synthesized 3D objects sampled from the learned model by Langevin
dynamics. For the last four synthesized objects (shown in columns 6, 7, 8, and 9), their nearest neighbors retrieved from the training set are shown in columns 10, 11, 12, and 13.
Experiment 2: 3D Object Recovery
We test the conditional 3D DescriptorNet on the 3D object recovery task.
Experiment 3: 3D Object Super resolution
We test the conditional 3D DescriptorNet on the 3D object super-resolution task
Figure3. 3D object super-resolution by conditional 3D DescriptorNet. The first row displays some original 3D objects (64×64×64 voxels). The second row shows the corresponding low resolution 3D objects (16 × 16 × 16 voxels). The last row displays the corresponding super-resolution results which are obtained by sampling from the conditional 3D DescriptorNet by running 10 steps of Langevin dynamics initialized with the objects shown in the second row
Experiment 4: Cooperative training of 3D generator
We evaluate a 3D generator trained by a 3D DescriptorNet
via MCMC teaching. We show results of interpolating between two latent vectors and shape arithmetic in the latent space.
Figure4. Interpolation between latent vectors of the 3D objects on the two ends.
3D Object Arithmetic
Figure5. 3D shape arithmetic in the latent space.
Experiment 5: 3D object classification
We evaluate the feature maps learned by our 3D DescriptorNet. We perform a classification experiment on ModelNet10 dataset. We first train a single model on all categories of the training set in an unsupervised manner. We train a multinomial
logistic regression classifier from labeled data based on the extracted feature vectors for classification.
This project builds upon the following ideas, which we encourage you to check out.
 Jianwen Xie*, Yang Lu*, Song-Chun Zhu, Ying Nian Wu. "A theory of generative convnet." ICML. 2016.
 Tian Han*, Yang Lu*, Song-Chun Zhu, Ying Nian Wu. "Alternating Back-Propagation for Generator Network." AAAI 2017.
 Jianwen Xie, Song-Chun Zhu, and Ying Nian Wu. "Synthesizing dynamic patterns by spatial-temporal generative ConvNet." CVPR . 2017.
 Jianwen Xie, Yang Lu, Ruiqi Gao, Ying Nian Wu. "Cooperative Learning of Energy-Based Model and Latent Variable Model via MCMC Teaching." AAAI. 2018.