Statistical Modeling and Learning
in Vision and Cognition

Stat 232A-CS266A, MW 2:00-3:15 pm, January-March, 2017, WGYoung 1044 [syllabus.pdf]

Course Description

This graduate level course introduces the principles, theories, and algorithms for modeling complex patterns in very high dimensional spaces, learning these statistical models from large data sets, and verifying them through stochastic sampling (synthesis). More specifically we study two classes of statistical models:

  1. Descriptive models (Markov random fields, Gibbs distributions, flat graphical models, multi-layered Neural Nets); and
  2. Generative models (sparse coding, auto-encoding, stochastic grammars, attributed grammars, hierarchical graphical models)

Their integration leads to a general unified theory for pusuing statistical models over a series of probabilistic families. The course also teaches a common framework for conceptualizing stochastic patterns and for statistical knowledge representation. Although the lectures will mostly focus on visual patterns in images and videos, and the methodology should be generally applicable to a wide range of applications, such as biologic patterns, network traffic modeling, material science, artificial intelligence, cognitive modeling, and autonomous robots, etc.

The visual patterns that we will study include:

  1. Spatial patterns --- primitives (textures and textons), parts, objects and scenes;
  2. Temporal patterns --- motion primitive, actions, events, group activities; and
  3. Causal patterns --- fluents and actions recursion.

We will study an information projection principle for learning these patterns and models in an unsupervised or weakly supervised way, as well as some theories of learning in the space of and-or graphs (AOG).

Prerequsites : Basic statistics, linear algebra, programming skills for a project. Knowledge and experience on images will be a plus.

Textbook [draft book pdf]


Grading Plan

Two homework assignments
HW1 [5 %] -- HW2 [5% ]


Small projects and exercises

  • 1. Natural image statistics, scale invariance [9%]
  • 2. image inpainting by MRF [6%]
  • 3. Sampling the Julesz texture ensemble [15%]
  • 4. Leaning FRAME model for reconstruction and synthesis [10%]
  • 5. Alternative back-propagation for hierarchical factor analysis [10]
Final Exam


List of Topics

  Chapter 1   Introduction to Knowledge Representation, Modeling and Learning                
   1. Towards a unified representation of commonsense knowledge
   2. Compositionality, reconfigurability, functionality and causality
   3. Representation of concepts   
   4. Examples and demos: Regimes of models

  Chapter 2  Empirical Observations: Image Space and Natural Image Statistics                           
   1. Empirical observation I: filtered responses
   2. Empirical observation II: scaling properties  
   3. Empirical observation III: patch frequency (structural and textural patches).
   4. Empirical observation IV:  information scaling and regimes of statistical models.
  Chapter 3  Descriptive Models I :  Classical Markov Random Fields                  
   1. Markov random field theory                                                        
   2. Gibbs fields: Ising and Potts models
   3. The equivalence of Gibbs and MRF (Hammersley-Clifford theorem)
   4. Early Markov random field models for images
   5, Maximum Entropy and Maximum Likelihood Eestimation
   6, Variations of likelihood: pseodo-, patch-, partial-likelihood
   7. From Gibbs distributions to PDEs in image processing

  Chapter 4 Descriptive Models II : FRAME Model and Julesz Ensemble                                   
   1. Pythagorean theorem and information projection                                        
   2. Minimax entropy learning and feature pursuit
   3. Julesz ensemble
   4. Ensemble equivalence theorem
   5. Ensembles in statistics mechanics 
   6. Examples on general prior, shape, curves, Gestalt field etc

  Chapter 5 Descriptive Models III : Hierarchical FRAME Model 
   1. Posing hierarchical Convolutional Neural Net as a unfolded Generalized Linear Model
   2. Defining hierarchical FRAME model
   3. Synthesis and reconstruction with hierarchical FRAME model 

  Chapter 6  Generative Models I:   Classical Models                                    
   1. Frame theory and wavelets                                              
   2. Design of frames:  image pyramids 
   3. Over-complete basis and matching pursuit   
   4. Markov tree and stochastic context free grammar

  Chapter 7  Generative Models II:   Sparse codeing, Textons, and Active Basis Models                                         
   1. Learning sparse coding from natural images
   2. Tangram models and hierarchical tiling
   3. Textons and image dictionary    
   4. Sparse representation of faces                                   
   5. Active basis model 
 Chapter 8  Generative Models III:  Hierarchical Factor Analysis   
   1. Factor analysis and auto-encoder
   2. Hierarchical Factor analysis with Convolutional Neural Nets
   3. Alternating back-propagation

  Chapter 9   Information Scaling, Scale Space and Imperceptibility   
   1. Image scaling and perceptual entropy
   2. A continuous entropy spectrum
   3. Perceptual scale space
   Chapter 10  Integrated Models: Descriptive + Generative                                
   1. Primal sketch model as low-middle representations for generic images and video                                  
   2, Hybrid image templates for objects and scenes
   3. An integrated language model(SCFG + bigram)
   4. And-Or graphs for context sensitive graph grammars

  Chapter 11   Learning Compositional Models 
   1. Unsupervised learning of And-Or graph: Block pursuit and bi-clustering.
   2. Learning temporal And-or graph (event grammar) for actions and events.
   3. Learning causal and-or graph for perceptual causality.
  Chapter 12  Discussion on Advanced Topics in Cognition and Robotics 
   1. Attribute AOG for human attributes
   2. Attribute AOG for single view 3D scene parsing
   3. Spatial, Temporal and Causal reasoning: using objects as tools
   3. Utility: loss and cost in human behaviors