Jianwen Xie 谢建文

• I am a researcher in artificial intelligence, with a variety of research experiences on machine learning, computer vision, and AI for science. I worked at Akool Research for AI-Generated Content, BioMap Research for AI-driven Drug Design (led by Prof. Le Song), Baidu Research for Fundamental Generative AI (led by Prof. Ping Li), Hikvision Research for Computer Vision, and UCLA Center for Vision, Cognition, Learning, and Autonomy for Explainable AI. I received my Ph.D. in Statistics at University of California, Los Angeles, under the supervision of Prof. Ying Nian Wu and Prof. Song-Chun Zhu. My primary research interest lies in Generative AI. I am particularly interested in generative modeling, unsupervised learning, Monte Carlo sampling, and representation learning. I am also an IEEE Senior member.

• [03/2023] I will serve as an Area Chair for NeurIPS 2024
• [01/2024] I will serve as an Area Chair for ACM Multimedia 2024
• [01/2024] I will serve as an Area Chair for ECCV 2024
• [12/2023] I will serve as an Area Chair for ICML 2024
• [11/2023] I will serve as a Senior Program Committee member for IJCAI 2024
• [08/2023] I will serve as an Area Chair for CVPR 2024 and ICLR 2024
• [06/2023] I will serve as a Senior Program Committee member for AAAI 2024

• 2010-2016
• University of California, Los Angeles, USA
Ph.D. in Statistics, 2016
Master of Science in Statistics, 2014
Master of Science in Computer Science, 2012

• 2004-2008
• Jinan University, China
• Bachelor of Engineering in Software Engineering, 2008

• Generative Models, AI for Science, Unsupervised Learning, Representation Learning

Associate Editor:

Area Chair:

Senior Program Committee:

• (Theme 1) Deep Energy-Based Model
• Energy-Based Generative ConvNet for Images and Theoretical Foundation (ICML 2016 by Xie et al. )
• Energy-Based Generative Spatial-Temporal ConvNet for Videos (CVPR 2017, TPAMI 2020 by Xie et al. )
• Energy-Based Generative VoxelNet for 3D Volumetric Shapes (CVPR 2018, TPAMI 2020 by Xie et al. )
• Energy-Based Generative PointNet for Unordered 3D Point Clouds (CVPR 2021 by Xie et al. )
• Energy-Based Patchwise Generative ConvNet for Deep Internal Learning (CVPR 2021 by Zheng et al. )
• Energy-Based Inverse Optimal Control for Trajectory Prediction (TNNLS 2022 by Xu et al. )
• Energy-Based Generative ConvNet for High-Fidelity Image Synthesis (ICLR 2021 by Zhao et al. )

Examples of energy-based synthesis of textures, objects, faces, scenes, videos, 3D voxels, unordered point clouds, trajectories, and unpaired image-to-image tanslation.

• (Theme 2) Generative Cooperative Learning
• Cooperative Networks (CoopNets) = EBM + Generator (AAAI 2018, TPAMI 2018 by Xie et al.)
• Conditional Cooperative Networks (CCoopNets) = Conditional EBM + Conditional Generator (TPAMI 2021 by Xie et al.)
• Cycle-Consistent Cooperative Networks (CycleCoopNets) + Alternating MCMC Teaching (AAAI 2021 by Xie et al.)
• Variational MCMC Teaching = EBM + VAE (AAAI 2021 by Xie et al.)
• Generative Cooperative Saliency Prediction (SalCoopNets) (AAAI 2022 by Zhang et al.)
• Cooperative Flow = Langevin Flow + Normalizing Flow (CoopFlow) (ICLR 2022 by Xie et al.)
• Initializing Adversarial Learning by Cooperative Learning (CoopInit) (AAAI 2023 by Zhao et al.)
• Cooperative Learning for Supervised Image Hashing (CoopHash) (arXiv 2022 by Doan et al.)
• Cooperative Diffusion Recovery Likelihood (ICLR 2024 by Zhu et al.)

Examples of cooperative learning on image synthesis, supervised conditional learning, unpaired image-to-image translation, saliency prediction, and sequence-to-sequence translation.

• (Theme 3) Deep Latent Variable Model
• Dynamic Generator + Alternating Back-propagation Through Time (AAAI 2019 by Xie et al.)
• Motion-Based Generator + Alternating Back-propagation Through Time (AAAI 2020 by Xie et al.)
• Feature-to-Feature Translator + Conditional Alternating Back-propagation (ICCV 2019 by Zhu et al.)
• Noise-Aware Encoder-Decoder + Conditional Alternating Back-propagation (ECCV 2020 by Zhang et al.)
• Short-Run MCMC Inference with Optimal Transport Correction (CVPR 2021 by An et al.)
• Generative Vision Transformer with Energy-Based Prior (NeurIPS 2021, TPAMI 2023 by Zhang et al.)
• Generator with Latent Space Flow-based Prior + Langevin Dynamics Inference (AAAI 2023 by Xie et al.)
• Generative Radiance Field with Energy-Based Prior (AISTATS 2023 by Zhu et al.)

• (Theme 4) Vector-Matrix Representational Model
• Grid Cell Model (ICLR 2019 by Gao et al.)
• Grid Cell Model with Head Direction (NeurIPS 2021 by Gao et al.)
• V1 Cell Model (AAAI 2022 by Gao et al.)



While vector can be used to represent entities (nouns), matrix can be used to represent actions, changes, and relations (verbs) that form groups. Group representation is a central topic in modern mathematics and physics.

• (Theme 5) FRAME (Filters, Random Fields and Maximum Entropy) Models
• Sparse Inhomogeneous FRAME Model (CVPR 2014, IJCV 2014 by Xie et al.)
• Gnerative Boosting for Sparse FRAME Model (ACHA 2015 by Xie et al.)
• Hierarchical Sparse FRAME Model (CVPR 2017 by Xie et al.)
• Deep FRAME Model (AMSA 2018 by Wu et al.)





Examples of image synthesis from the FRAME models via MCMC.

• 1. CVPR 2021 Tutorial on Theory and Application of Energy-Based Generative Models
• --Jianwen Xie, Ying Nian Wu


• 2. ICCV 2021 Tutorial on Theory and Application of Energy-Based Generative Models
• --Jianwen Xie, Ying Nian Wu


• 3. IJCAI 2022 Tutorial on Deep Energy-Based Learning
• --Jianwen Xie


• 4. ECCV 2022 Tutorial on Deep Energy-Based Learning in Computer Vision
• --Jianwen Xie

• Generative Modeling and Unsupervised Learning in Computer Vision, University of California, Los Angeles, 2016

Latent Plan Transformer: Planning as Latent Variable Inference [PDF]
• --Deqian Kong, Dehong Xu, Minglu Zhao, Bo Pang, Jianwen Xie, Andrew Lizarraga, Yuhao Huang, Sirui Xie, Ying Nian Wu
• --arXiv 2024


• Dual-Space Optimization: Improved Molecule Sequence Design by Latent Prompt Transformer [PDF]
• --Deqian Kong, Yuhao Huang, Jianwen Xie, Edouardo Honig, Ming Xu, Shuanghong Xue, Pei Lin, Sanping Zhou. Sheng Zhong, Nanning Zheng, Ying Nian Wu
• --arXiv 2024
• (an expanded version of the NeurIPS-23 workshop paper [PDF] on AI for science)


• Progressive Energy-Based Cooperative Learning for Multi-Domain Image-to-Image Translation [PDF]
• --Weinan Song, Yaxuan Zhu, Lei He, Ying Nian Wu, Jianwen Xie
• --arXiv 2024


• CoopHash: Cooperative Learning of Multipurpose Descriptor and Contrastive Pair Generator via Variational MCMC Teaching for Supervised Image Hashing [PDF]
• --Khoa Doan, Jianwen Xie, Yaxuan Zhu, Yang Zhao, Ping Li
• --arXiv 2024


• Learning Energy-Based Models by Cooperative Diffusion Recovery Likelihood [PDF]
• --Yaxuan Zhu, Jianwen Xie, Ying Nian Wu, Ruiqi Gao
• --The Twelfth International Conference on Learning Representations (ICLR) 2024


• An Energy-Based Prior for Generative Saliency [PDF]
• --Jing Zhang, Jianwen Xie, Nick Barnes, Ping Li
• --IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI) 2023
• (an expanded version of the NeurIPS-21 paper for generative saliency)


• Likelihood-Based Generative Radiance Field with Latent Space Energy-Based Model for 3D-Aware Disentangled Image Representation [PDF] [code]
• --Yaxuan Zhu, Jianwen Xie, Ping Li
• --The Twenty-Sixth International Conference on Artificial Intelligence and Statistics (AISTATS) 2023


• A Tale of Two Latent Flows: Learning Latent Space Normalizing Flow with Short-run Langevin Flow for Approximate Inference [PDF] [project page]
• --Jianwen Xie, Yaxuan Zhu, Yifei Xu, Dingcheng Li, Ping Li
• --The Thirty-Seventh AAAI Conference on Artificial Intelligence (AAAI) 2023


• CoopInit: Initializing Generative Adversarial Networks via Cooperative Learning [PDF]
• --Yang Zhao, Jianwen Xie, Ping Li
• --The Thirty-Seventh AAAI Conference on Artificial Intelligence (AAAI) 2023


• Energy-Based Continuous Inverse Optimal Control [PDF] [project page]
• --Yifei Xu, Jianwen Xie, Tianyang Zhao, Chris Baker, Yibiao Zhao, Ying Nian Wu
• --IEEE Transactions on Neural Networks and Learning Systems (TNNLS) 2022
• (an expanded version of the NeurIPS 2020 workshop paper on machine learning for autonomous driving)


• Blind Image Super-Resolution with Elaborate Degradation Modeling on Noise and Kernel [PDF]
• --Zongsheng Yue, Qian Zhao, Jianwen Xie, Lei Zhang, Deyu Meng, Kwan-Yee K. Wong
• --IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 2022


• A Tale of Two Flows: Cooperative Learning of Langevin Flow and Normalizing Flow Toward Energy-Based Model [PDF] [project page]
• --Jianwen Xie, Yaxuan Zhu, Jun Li, Ping Li
• --Tenth International Conference on Learning Representations (ICLR) 2022


• Learning V1 Simple Cells with Vector Representations of Local Contents and Matrix Representations of Local Motions [PDF] [project page]
• --Ruiqi Gao, Jianwen Xie, Siyuan Huang, Yufan Ren, Song-Chun Zhu, Ying Nian Wu
• --The Thirty-Sixth AAAI Conference on Artificial Intelligence (AAAI) 2022


• Energy-Based Generative Cooperative Saliency Prediction [PDF]
• --Jing zhang, Jianwen Xie, Zilong Zheng, Nick Barnes
• --The Thirty-Sixth AAAI Conference on Artificial Intelligence (AAAI) 2022


• On Path Integration of Grid Cells: Group Representation and Isotropic Scaling [PDF]
• --Ruiqi Gao, Jianwen Xie, Xue-Xin Wei, Song-Chun Zhu, Ying Nian Wu
• --Thirty-fifth Annual Conference on Neural Information Processing Systems (NeurIPS) 2021


• Learning Generative Vision Transformer with Energy-Based Latent Space for Saliency Prediction [PDF]
• --Jing Zhang, Jianwen Xie, Nick Barnes, Ping Li
• --Thirty-fifth Annual Conference on Neural Information Processing Systems (NeurIPS) 2021


• Monocular 3D Pose Estimation via Pose Grammar and Data Augmentation [PDF]
• --Yuanlu Xu, Wenguan Wang, Tengyu Liu, Xiaobai Liu, Jianwen Xie, Song-Chun Zhu
• --IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI) 2021


• Cooperative Training of Fast Thinking Initializer and Slow Thinking Solver for Conditional Learning [PDF] [project page]
• --Jianwen Xie *, Zilong Zheng *, Xiaolin Fang, Song-Chun Zhu, Ying Nian Wu (* equal contributions)
• --IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI) 2021


• Patchwise Generative ConvNet: Training Energy-Based Models from a Single Natural Image for Internal Learning [PDF] [project page]
• --Zilong Zheng, Jianwen Xie, Ping Li
• --IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 2021 (Oral)


• Learning Deep Latent Variable Models by Short-Run MCMC Inference with Optimal Transport Correction [PDF]
• --Dongsheng An, Jianwen Xie, Ping Li
• --IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 2021


• Generative PointNet: Deep Energy-Based Learning on Unordered Point Sets for 3D Generation, Reconstruction and Classification [PDF] [project page]
• --Jianwen Xie *, Yifei Xu *, Zilong Zheng, Song-Chun Zhu, Ying Nian Wu (* equal contributions)
• --IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 2021


• Semi-Supervised Video Deraining with Dynamical Rain Generator [PDF]
• --Zongsheng Yue, Jianwen Xie, Qian Zhao, Deyu Meng
• --IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 2021


• Learning Energy-Based Generative Models via Coarse-to-Fine Expanding and Sampling [PDF] [code]
• --Yang Zhao, Jianwen Xie, Ping Li
• --Ninth International Conference on Learning Representations (ICLR) 2021


• Learning Cycle-Consistent Cooperative Networks via Alternating MCMC Teaching for Unsupervised Cross-Domain Translation [PDF] [project page]
• --Jianwen Xie *, Zilong Zheng *, Xiaolin Fang, Song-Chun Zhu, Ying Nian Wu (* equal contributions)
• --The Thirty-Fifth AAAI Conference on Artificial Intelligence (AAAI) 2021


• Learning Energy-Based Model with Variational Auto-Encoder as Amortized Sampler [PDF] [project page]
• --Jianwen Xie, Zilong Zheng, Ping Li
• --The Thirty-Fifth AAAI Conference on Artificial Intelligence (AAAI) 2021


• Generative VoxelNet: Learning Energy-Based Models for 3D Shape Synthesis and Analysis [PDF] [project page]
• --Jianwen Xie *, Zilong Zheng *, Ruiqi Gao, Wenguan Wang, Song-Chun Zhu, Ying Nian Wu (* equal contributions)
• --IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI) 2020
• (an expanded version of the CVPR-18 paper for energy-based 3D descriptor network)


• Learning Noise-Aware Encoder-Decoder from Noisy Labels by Alternating Back-Propagation for Saliency Detection [PDF]
• --Jing Zhang, Jianwen Xie, Nick Barnes
• --European Conference on Computer Vision (ECCV) 2020


• Motion-Based Generator Model: Unsupervised Disentanglement of Appearance, Trackable and Intrackable Motions in Dynamic Patterns [PDF][project page]
• --Jianwen Xie *, Ruiqi Gao *, Zilong Zheng, Song-Chun Zhu, Ying Nian Wu (* equal contributions)
• --The Thirty-Fourth AAAI Conference on Artificial Intelligence (AAAI) 2020 (Oral)


• Representation Learning: A Statistical Perspective [PDF][Tex]
• --Jianwen Xie, Ruiqi Gao, Erik Nijkamp, Song-Chun Zhu, Ying Nian Wu
• --Annual Review of Statistics and Its Application (ARSIA) 2020
• (an review article on the subject of Representation Learning to Annual Reviews, whose Impact Factor rankings are among the highest in scholarly publishing)
• 
  
• Learning Energy-based Spatial-Temporal Generative ConvNet for Dynamic Patterns [PDF] [project page]
• --Jianwen Xie, Song-Chun Zhu, Ying Nian Wu
• --IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI) 2019
• (an expanded version of the CVPR-17 paper for spatial-temporal generative ConvNet)


• Revisiting Video Saliency Prediction in the Deep Learning Era [PDF]
• --Wenguan Wang, Jianbing Shen, Jianwen Xie, Ming-Ming Cheng, Haibin Ling, Ali Borji
• --IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI) 2019


• Semantic-Guided Multi-Attention Localization for Zero-Shot Learning [PDF]
• --Yizhe Zhu, Jianwen Xie, Zhiqiang Tang, Xi Peng, Ahmed Elgammal
• --Thirty-third Conference on Neural Information Processing Systems (NeurIPS) 2019


• Learning Feature-to-Feature Translator by Alternating Back-Propagation for Generative Zero-Shot Learning [PDF]
• --Yizhe Zhu, Jianwen Xie, Bingchen Liu, Ahmed Elgammal
• --International Conference on Computer Vision (ICCV) 2019


• Learning Grid Cells as Vector Representation of Self-Position Coupled with Matrix Representation of Self-Motion [PDF][project page]
• --Ruiqi Gao *, Jianwen Xie *, Song-Chun Zhu, Ying Nian Wu (* equal contributions)
• --Seventh International Conference on Learning Representations (ICLR) 2019


• Learning Dynamic Generator Model by Alternating Back-Propagation Through Time [PDF][project page]
• --Jianwen Xie *, Ruiqi Gao *, Zilong Zheng, Song-Chun Zhu, Ying Nian Wu (* equal contributions)
• --The Thirty-Third AAAI Conference on Artificial Intelligence (AAAI) 2019 (Spotlight)


• Segregated Temporal Assembly Recurrent Networks for Weakly Supervised Multiple Action Detection
• --Yunlu Xu, Chengwei Zhang, Zhanzhan Cheng, Jianwen Xie, Yi Niu, Shiliang Pu, Fei Wu
• --The Thirty-Third AAAI Conference on Artificial Intelligence (AAAI) 2019 (Oral)


• Cooperative Training of Descriptor and Generator Networks [PDF][project page]
• --Jianwen Xie, Yang Lu, Ruiqi Gao, Song-Chun Zhu, Ying Nian Wu
• --IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI) 2018
• (an expanded version of the AAAI-18 paper for cooperative learning)


• Learning Descriptor Networks for 3D Shape Synthesis and Analysis [PDF][project page]
• --Jianwen Xie *, Zilong Zheng *, Ruiqi Gao, Wenguan Wang, Song-Chun Zhu, Ying Nian Wu (* equal contributions)
• --IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 2018 (Oral)


• A Causal And-Or Graph Model for Visibility Fluent Reasoning in Tracking Interacting Objects [PDF]
• --Yuanlu Xu *, Lei Qin *, Xiaobai Liu, Jianwen Xie, Song-Chun Zhu (* equal contributions)
• --IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 2018


• Weakly and Semi Supervised Human Body Part Parsing via Pose-Guided Knowledge Transfer [PDF]
• --Hao-Shu Fang, Guansong Lu, Xiaolin Fang, Jianwen Xie, Yu-Wing Tai, Cewu Lu
• --IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 2018 (Spotlight)


• Cooperative Learning of Energy-Based Model and Latent Variable Model via MCMC Teaching [PDF][project page]
• --Jianwen Xie, Yang Lu, Ruiqi Gao, Ying Nian Wu
• --The Thirty-Second AAAI Conference on Artificial Intelligence (AAAI) 2018 (Oral)


• Sparse and Deep Generalizations of the FRAME Model [PDF][Tex]
• --Ying Nian Wu, Jianwen Xie, Yang Lu, Song-Chun Zhu
• --Annals of Mathematical Sciences and Applications (AMSA) 2018
• (an invited paper to the AMSA special issue in honor of Professor David Mumford on his 80th birthday)


• Super-Trajectory for Video Segmentation [PDF]
• --Wenguan Wang, Jianbing Shen, Jianwen Xie, Fatih Porikli
• --International Conference on Computer Vision (ICCV) 2017


• Synthesizing Dynamic Pattern by Spatial-Temporal Generative ConvNet [PDF][project page]
• --Jianwen Xie, Song-Chun Zhu, Ying Nian Wu
• --IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 2017


• Generative Hierarchical Learning of Sparse FRAME Models [PDF][project page]
• --Jianwen Xie, Yifei Xu, Erik Nijkamp, Ying Nian Wu, Song-Chun Zhu
• --IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 2017


• Decoding the Encoding of Functional Brain Networks: an fMRI Classification Comparison of Non-negative Matrix Factorization (NMF), Independent Component Analysis (ICA), and Sparse Coding Algorithms [PDF]
• --Jianwen Xie, Pamela K. Douglas, Ying Nian Wu, Arthur L. Brody, Ariana E. Anderson
• --Journal of Neuroscience Methods 2017


• A Theory of Generative ConvNet [PDF]
• --Jianwen Xie *, Yang Lu *, Song-Chun Zhu, Ying Nian Wu (* equal contributions)
• (an expanded version of the ICML-16 paper written for statisticians.)


• A Theory of Generative ConvNet [PDF][project page]
• --Jianwen Xie *, Yang Lu *, Song-Chun Zhu, Ying Nian Wu (* equal contributions)
• --International Conference on Machine Learning (ICML) 2016 (Oral)


• Inducing Wavelets into Random Fields via Generative Boosting [PDF][project page]
• --Jianwen Xie, Yang Lu, Song-Chun Zhu, Ying Nian Wu
• --Journal of Applied and Computational Harmonic Analysis (ACHA) 2015
• (ACHA is a premier journal in harmonic analysis and applied mathematics)


• Learning Sparse FRAME Models for Natural Image Patterns [PDF][project page]
• --Jianwen Xie, Wenze Hu, Song-Chun Zhu, Ying Nian Wu
• --International Journal of Computer Vision (IJCV) 2014


• Learning Inhomogeneous FRAME Models for Object Patterns [PDF][project page]
• --Jianwen Xie, Wenze Hu, Song-Chun Zhu, Ying Nian Wu
• --IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 2014


• Non-negative Matrix Factorization of Multimodal MRI, fMRI and Phenotypic Data reveals Differential Changes in Default Mode Subnetworks in ADHD [PDF]
• --Ariana Anderson, Pamela Douglas, Wesley Kerr, Virginia Haynes, Alan Yuille, Jianwen Xie, Ying Nian Wu, Jesse Brown, Mark Cohen
• --NeuroImage 2014.