Stat 232A-CS266A:
 Statistical Modeling and Learning
 in Vision and Image Science

   TR 4:15-5:35 pm January-March 2008,
   Math Sci. Bldg 5137

Course Description

This graduate level course introduces the principles, theories, and algorithms for modeling complex patterns in very high dimensional spaces, learning these statistical models from large data sets, and verifying them through stochastic sampling (synthesis). More specifically we shall study two classes of statistical models,

• Descriptive models (Markov random fields, graphical models)
• Generative models (wavelet, sparse coding, stochastic grammars)

We shall discuss the variants of these models and their integrations, which leads to a general unified theory for augmenting statistical models over a series of probabilistic families. The course also teaches a common framework for conceptualizing stochastic patterns and for statistical knowledge representation.

Although the lectures will mostly focus on visual patterns in images, the methodology should be generally applicable to a wide range of applications, such as biologic patterns, network traffic modeling, etc.

Prerequisites

• Basic statistics, linear algebra, programming skills for a project.
• Knowledge and experience on images will be a plus

Textbook

• There is no textbook. Teaching notes and papers will be distributed in this web page.

Instructor

• Prof. Song-Chun Zhu, sczhu@stat.ucla.edu, 310-206-8693, office BH 9404. Office Hours: Monday 3:30-5pm

Grading Plan: 

    The grade will be based on four parts
       1. class participation and discussion            10%
       2. 2-3 homework assignments                    30%
       3. 1 reading assignment and presentation    30%
       4. final exam                                              30%

      Each student  can choose a set of advanced papers to read, simulate some models,  and present (15 minutes)
the results. The grade on this part will be based on the difficulties of the model and the quality of the sampled results.
    Topics of projects will be discussed between students and instructor ( a suggested list of projects is below).

List of Topics

List of references, List of projects, on-line reading materials syllabus.pdf

  Chapter 1   Introduction                                                          [ch1.pdf]
   1. Knowledge discovery and representation by statistical modeling 
   2. The spaces of data and the spaces of patterns
   3. Formulation of statistical modeling
   4. Principles, thoughts, and research streams
       (regularity, redundancy reduction, manifolds, coding, entropy, MLE, sparsity)

 Chapter 2  Empirical statistical observations in image data                               [ch2.pdf] 
   1. Empirical observation I:  filtered responses
         (High-kurtosis, Generalized Gaussian, and Cauchy)
   2. Empirical observation II:  scaling properties  
         (the 1/f power-law, scale invariance of gradients, and entropy rate over scales.)
   3. Empirical observation III: patch frequency
         (manifolds)
   4. Empirical observation IV: lighting and motion manifolds

  Chapter 3  Descriptive models I  :  classical MRF                                      [ch3_fig.pdf] 
   1. Markov random field theory                                    [Read Winkler Chapter]
         (neighborhoods, cliques and potentials)                             
   2. Gibbs fields
         (Ising and Potts models)
   3. The equivalence of Gibbs and MRF
         (Hammersley-Clifford theorem)
   4. Early Markov random field models for images
         (Weak membrane, regularization, line process, Mumford-Shah, total variance)
   5. An extra session (?): Basic Markov chain Monte Carlo techniques for model simulation 
         (Gibbs, exact sampling etc.)

  Chapter 4 Descriptive models II : advanced                                          [ch4_fig.pdf]
   1. Maximum entropy learning                                        
   2. Minimax entropy learning and feature pursuit
         (Texture examples)
   3. Julesz ensemble
   4. Ensemble equivalence theorem
   5. Ensembles in statistics mechanics 
   6. Conceptualization of stochastic patterns
   7. Examples on general prior, shape, curves, Gestalt field, face, scene labels etc
   8. A general picture of descriptive learning
         (Modeling in general graph spaces)
        
  Chapter 5  Descriptive models III :  variants                                           [ch5.pdf]
   1. Causal Markov models, non-parametric methods
          (texture models, expansion and compression)
   2. pseudo-descriptive models
          (pyramidal sampling and collapsing)
   3. pseudo-likelihood, patch/partial likelihood
   3. Mixed Markov Fields
          (dynamic neighborhood, hard core model)

  Chapter 6  Generative models I:   classical models                                     [ch6-7.pdf]
   1. Frame theory
          (frame, tight frame, pseudo-inverse)
   2. Design of frames:  image pyramids 
          (Gaussian, Laplacian, Gabor, Steerable)
   3. Over-complete basis and matching pursuit
   4. Sparsity definition
          (numerosity, decay rate and critical index)
   5. Computational Harmonic analysis
          (functional spaces)
   6. Markov tree and stochastic context free grammar

  Chapter 7  Generative models II:  advanced                                         
   1. Learning sparse coding from natural images
   2. random wavelets, dead-leaves, random collage, and cemetery models.
          (revisit scale invariance)
   3. Texton models
          (textons, lightons, and active basis)
   4. Kolmogorov epsilon-entropy 
          (data ensemble, capacity dimension, theorem)
   5. The origin of hidden variables
          (revisit K-mean clustering)
   6. Implicit manifold vs. explicit manifold learning:
          (the transition of the two classes of models)
      
 Chapter 8   Integrated models: descriptive + generative                                 [Ch8.pdf]
   1. An integrated language model
                (SCFG + MRF)  
   2. Primal sketch model 
                (Structures + textures)
   3. And-Or graphs for context sensitive graph grammars
   4. Composite and reconfurable templates
        (clothes modeling and functional object class modeling)
   5. Examples in object recognition

  Chapter 9  Conclusion and discussion : a grand unifying theory                                                    [Ch9.pdf]
      1. Statistical modeling and conceptualization 
   2. Manifold pursuit in the image space.
   3. A unified learning framework for descriptive, generative and discriminative
       model pursuit.