\documentclass[10pt,twocolumn,letterpaper]{article}

\usepackage{cvpr}
\usepackage{times}
\usepackage{epsfig}
\usepackage{graphicx}
\usepackage{amsmath}
\usepackage{amssymb}


\usepackage[utf8]{inputenc} % allow utf-8 input
\usepackage[T1]{fontenc}    % use 8-bit T1 fonts
%\usepackage{hyperref}       % hyperlinks
\usepackage{url}            % simple URL typesetting
\usepackage{booktabs}       % professional-quality tables
\usepackage{amsfonts}       % blackboard math symbols
\usepackage{nicefrac}       % compact symbols for 1/2, etc.
\usepackage{microtype}      % microtypography
%\usepackage[pdftex]{graphicx}
\usepackage{algorithm}
\usepackage{algorithmic}
%\usepackage{amsmath}
%\usepackage{amssymb}
%\usepackage{mathrsfs}
\usepackage{array}
\usepackage{bm}
\usepackage{multirow}
%\usepackage{natbib}

\def\obs{{\rm obs}}
\def\syn{{\rm syn}}

\renewcommand{\algorithmicrequire}{\textbf{Input:}}
\renewcommand{\algorithmicensure}{\textbf{Output:}}



% Include other packages here, before hyperref.

% If you comment hyperref and then uncomment it, you should delete
% egpaper.aux before re-running latex.  (Or just hit 'q' on the first latex
% run, let it finish, and you should be clear).
\usepackage[pagebackref=true,breaklinks=true,letterpaper=true,colorlinks,bookmarks=false]{hyperref}

 \cvprfinalcopy % *** Uncomment this line for the final submission

\def\cvprPaperID{3327} % *** Enter the CVPR Paper ID here
\def\httilde{\mbox{\tt\raisebox{-.5ex}{\symbol{126}}}}

% Pages are numbered in submission mode, and unnumbered in camera-ready
\ifcvprfinal\pagestyle{empty}\fi

\def\I{{\bf I}}
\def\B{{\bf B}}
\def\b{{\bf b}}
\def\a{\alpha}
\def\D{{\cal D}}
\def\S{{\cal S}}

\def\E{{\rm E}}

\def\tI{\tilde{\bf I}}
\def\hI{\hat{\bf I}}
\def\tM{\tilde{M}}
\def\F{{\bf F}}

\def\x{\hat{x}}
\def\k{\hat{k}}

\def\w{{\bf w}}





\begin{document}

%%%%%%%%% TITLE
\title{Synthesizing Dynamic Patterns by Spatial-Temporal Generative ConvNet}

\author{Jianwen Xie, Song-Chun Zhu, and Ying Nian Wu\\
University of California, Los Angeles (UCLA), USA\\
{\tt\small jianwen@ucla.edu, sczhu@stat.ucla.edu, ywu@stat.ucla.edu}}

\maketitle
%\thispagestyle{empty}

%%%%%%%%% ABSTRACT
\begin{abstract}
Video sequences contain rich dynamic patterns, such as dynamic texture patterns that exhibit stationarity in the temporal domain, and action  patterns that are non-stationary in either spatial or temporal domain. We show that a spatial-temporal generative ConvNet can be used to model and synthesize  dynamic patterns. The model defines a probability distribution on the video sequence, and the log probability is defined by a spatial-temporal ConvNet that consists of multiple layers of spatial-temporal filters to capture spatial-temporal patterns of different scales. The model can be learned from the training video sequences by an ``analysis by synthesis'' learning algorithm that iterates the following two steps. Step 1 synthesizes video sequences from the currently learned model. Step 2 then updates the model parameters based on the difference between the synthesized video sequences and the observed training sequences. We show that the learning algorithm can synthesize realistic dynamic patterns. 
%We also show that it is possible to learn the model from incomplete training sequences with either occluded pixels or missing frames, so that model learning and pattern completion can be accomplished simultaneously. 
  \end{abstract}

%%%%%%%%% BODY TEXT
\section{Introduction}

There are a wide variety of dynamic patterns in video sequences, including dynamic textures \cite{doretto2003dynamic} or textured motions \cite{wang2002generative} that  exhibit statistical stationarity or stochastic repetitiveness in the temporal dimension, and action  patterns that are non-stationary in either spatial or temporal domain. Synthesizing and analyzing such dynamic patterns has been an interesting problem. In this paper, we focus on the task of synthesizing dynamic patterns using a generative version of the convolutional neural network (ConvNet or CNN). 

The ConvNet \cite{lecun1998gradient, krizhevsky2012imagenet}  has proven to be an immensely successful discriminative learning machine. The convolution operation in the ConvNet is particularly suited for signals such as images, videos and sounds that exhibit translation invariance either in the spatial domain or the temporal domain or both. Recently, researchers have become increasingly interested in the generative aspects of ConvNet,  for the purpose of visualizing the knowledge learned by the ConvNet, or synthesizing realistic signals, or developing generative models that can be used for unsupervised learning. 

In terms of synthesis,  various approaches based on the ConvNet have been proposed to synthesize realistic static images \cite{Alexey2015, KarolICML2015, Denton2015a, Kulkarni2015, LuZhuWu2016}. However, there has not been much work in the literature on synthesizing dynamic patterns based on the ConvNet, and this is the focus of the present paper. 

Specifically, we propose to synthesize dynamic patterns by generalizing the generative ConvNet model recently proposed by \cite{XieLuICML}. The generative ConvNet can be derived from the discriminative ConvNet. It is a random field model or an energy-based model \cite{lecun2006tutorial, Ng2011} that is in the form of exponential tilting of a reference distribution such as the Gaussian white noise distribution or the uniform distribution. The exponential tilting is parametrized by a ConvNet that involves multiple layers of linear filters and rectified linear units (ReLU)  \cite{krizhevsky2012imagenet}, which seek to capture features or patterns at different scales. 
%The log probability density or the energy function of the model is the sum of a ConvNet term that is piecewise linear due to the ReLU non-linearity \cite{montufar2014number} and the $\ell_2$ norm of the signal that comes from the Gaussian white noise reference distribution. As a result, the energy function is piecewise quadratic, and the generative ConvNet model is piecewise Gaussian. Moreover, the local modes of the distribution is auto-encoding, where the auto-encoding process involves a bottom-up pass that computes the filter responses followed by a top-down pass that reconstructs the signal where the multiple layers of filters in the bottom-up pass serve as the basis functions in the top-down pass. Such an explicit representation is unique among energy-based models  \cite{lecun2006tutorial, Ng2011}, and is a result of the fusion between the ReLU piecewise linear structure and the $\ell_2$ norm term from the Gaussian white noise reference distribution. 

The generative ConvNet can be sampled by the Langevin dynamics. 
%Because of the aforementioned auto-encoding structure of the generative ConvNet, the Langevin dynamics is driven by the reconstruction error, i.e., the difference between the current sample and its reconstruction by the above auto-encoding process. 
The model can be learned by the stochastic gradient algorithm  \cite{younes1999convergence}. It is an ``analysis by synthesis'' scheme that seeks to match the synthesized signals generated by the Langevin dynamics to the observed training signals.  Specifically, the learning algorithm iterates the following two steps after initializing the parameters and the synthesized signals. Step 1 updates the synthesized signals by the Langevin dynamics that samples from the currently learned model. Step 2 then updates the parameters based on the difference between the synthesized data and the observed data in order to shift the density of the model from the synthesized data towards the observed data. It is shown by  \cite{XieLuICML}  that the learning algorithm can synthesize realistic spatial image patterns such as textures and objects. 

In this article, we generalize the spatial generative ConvNet by adding the temporal dimension, so that the resulting ConvNet consists of multiple layers of spatial-temporal filters that seek to capture spatial-temporal patterns at various  scales. 
%For dynamic textures, these spatial-temporal filters are convolutional in the temporal domain, reflecting the statistical stationarity or stochastic repetitiveness of the patterns in the temporal domain. If the dynamic textures also exhibit spatial stationarity, we also make the spatial-temporal filters convolutional in the spatial domain. For action and motion patterns that are non-stationary in the temporal domain, the top layer filters are fully connected in the temporal domain. 
We show that the learning algorithm for training the spatial-temporal generative ConvNet can synthesize realistic dynamic patterns.  We also show that it is possible to learn the model from incomplete video sequences with either occluded pixels or missing frames, so that model learning and pattern completion can be accomplished simultaneously. 


\section{Related work} 

Our work is a generalization of the generative ConvNet model of \cite{XieLuICML} by adding the temporal dimension. \cite{XieLuICML} did not work on dynamic patterns such as those in the video sequences. The spatial-temporal discriminative ConvNet was used by \cite{ji20133d} for analyzing video data. The connection between discriminative ConvNet and generative ConvNet was studied by \cite{XieLuICML}. 


Dynamic textures or textured motions have been studied by \cite{doretto2003dynamic,wang2002generative,wang2004analysis, han2015video}. For instance,  \cite{doretto2003dynamic} proposed a vector auto-regressive model coupled with frame-wise dimension reduction by single value decomposition. It is a linear model with Gaussian innovations. \cite{wang2002generative} proposed a dynamic model based on sparse linear representation of frames. See \cite{you2016kernel} for a recent review of dynamic textures. The spatial-temporal generative ConvNet is a non-linear and non-Gaussian model and is expected to be more flexible in capturing complex spatial-temporal patterns in dynamic textures with multiple layers of non-linear spatial-temporal filters. 

Recently \cite{vondrick2016generating} generalized the generative adversarial networks \cite{goodfellow2014generative} to model dynamic patterns. Our model is an energy-based model and it also has an adversarial interpretation. See section \ref{sect:adv} for details. 

For temporal data, a popular model is the recurrent neural network \cite{williams1989learning, hochreiter1997long}. It is a causal model and it requires a starting frame. In contrast, our model is non-causal, and does not require a starting frame. Compared to the recurrent network, our model is more convenient and direct in capturing temporal patterns at multiple time scales. 


\section{Spatial-temporal generative ConvNet}
\label{gen_inst}


%\begin{figure}
%\begin{center}
%\includegraphics[width=.96\linewidth]{./illustration/ST_CNN.jpg}
%\caption{Spatial-temporal generative ConvNet consists of multiple layers of spatial-temporal filtering and sub-sampling operations for bottom-up feature extraction, resulting in multiple layers of feature maps (in yellow) and their sub-sampled versions (in blue). This figure illustrates an example of 2 layer network, where green cubes represent spatial-temporal filters. No pooling layers are used, but spatially fully connected layers are allowed.}
%\label{fig:illustration}
%\end{center}
%\end{figure}

\subsection{Spatial-temporal filters}

To fix notation, let $\I(x, t)$ be an image sequence of a video defined on the square (or rectangular) image domain $\D$ and the time domain ${\cal T}$, where $x = (x_1, x_2) \in \D$ indexes the coordinates of pixels, and $t \in {\cal T}$ indexes the frames in the video sequence. We can treat $\I(x, t)$ as a three dimensional function defined on $\D \times {\cal T}$.  For a spatial-temporal filter $F$,  we let $F*\I$ denote the filtered image sequence or feature map, and let $[F*\I](x, t)$ denote the filter response or feature at pixel $x$ and time $t$. 

The spatial-temporal ConvNet is a composition of multiple layers of linear filtering and ReLU non-linearity, as expressed by the following recursive formula: 
\begin{equation}
\begin{aligned}
 & [F^{(l)}_{k}  *\I] (x, t)  =    h\Bigg(\sum_{i=1}^{N_{l-1}}  \sum_{(y, s) \in \S_{l}} w^{(l, k)}_{i, y, s}    \\
 &  \times  [F^{(l-1)}_{i}*\I](x+y, t + s) + b_{l, k}\Bigg), 
\end{aligned}
\label{eq:ConvNet}
\end{equation}
where $l \in \{1, 2, ..., {L}\}$ indexes the layers.  $\{F^{(l)}_k, k = 1, ..., N_l\}$ are the filters at layer $l$, and $\{F^{(l-1)}_i, i = 1, ..., N_{l-1}\}$ are the filters at layer $l-1$.  $k$ and $i$ are used to index filters at layers $l$ and $l-1$ respectively, and $N_l$ and $N_{l-1}$ are the numbers of filters at layers $l$ and $l-1$ respectively. The filters are locally supported, so the range of $(y, s)$  is within a local support $\S_{l}$  (such as a $7 \times 7 \times 3$ box of image sequence). The weight parameters $(w^{(l, k)}_{i, y, s}, (y, s)  \in \S_l, i = 1, ..., N_{l-1})$ define a linear filter that operates on $(F^{(l-1)}_{i}*\I, i = 1, ..., N_{l-1})$. The linear filtering operation is followed by ReLU $h(r) = \max(0, r)$.  At the bottom layer, $[F^{(0)}_k*\I](x, t) = \I_k(x, t)$, where $k \in \{{\rm R, G, B}\}$ indexes the three color channels. Sub-sampling may be implemented so that in  $[F^{(l)}_{k}  *\I](x, t)$, $x \in \D_l \subset \D$, and $t \in {\cal T}_l \subset {\cal T}$.  %See Figure \ref{fig:illustration} for an illustration. 

The spatial-temporal filters at multiple layers are expected to capture the spatial-temporal patterns at multiple scales. It is possible that the top-layer filters are fully connected in the spatial domain as well as the temporal domain (e.g., the feature maps are $1 \times 1$ in the spatial  domain) if the dynamic pattern does not exhibit spatial or temporal stationarity. 



\subsection{Spatial-temporal generative ConvNet} 

The spatial-temporal generative ConvNet is an energy-based model or a random field model  defined on the image sequence $\I = (\I(x, t), x \in {\cal D}, t \in {\cal T})$. It is in the form of exponential tilting of a reference distribution $q(\I)$: 
\begin{equation}
p(\I; w) = \frac{1}{Z(w)} \exp \left[ f(\I; w)\right] q(\I), 
\label{eq:ConvNet-FRAME}
\end{equation}
where the scoring function $f(\I; w)$ is 
\begin{equation}
f(\I; w) =\sum_{k=1}^{K} \sum_{x \in {\cal D}_L} \sum_{t \in {\cal T}_L} [F_k^{(L)}*\I](x, t),  
\label{eq:scoring}
\end{equation}
where $w$ consists of all the weight and bias terms that define the filters $(F_k^{(L)}, k = 1, ..., K = N_L)$ at layer $L$, and $q$ is the Gaussian white noise model, i.e., 
\begin{equation}
q(\I) = \frac{1}{(2\pi\sigma^2)^{|{\cal D} \times {\cal T}|/2}} \exp\left[- \frac{1}{2\sigma^2} ||\I||^2\right], 
\label{eq:Gaussian}
\end{equation}   
where $|\D \times {\cal T}|$ counts the number of pixels in the domain $\D \times {\cal T}$. Without loss of generality, we shall assume $\sigma^2 = 1$. 

The scoring function $f(\I; w)$ in (\ref{eq:scoring}) tilts the Gaussian reference distribution into a non-Gaussian model. In fact, the purpose of $f(\I; w)$ is to identify the non-Gaussian spatial-temporal features or patterns. In the definition of $f(\I; w)$ in (\ref{eq:scoring}), we sum over the filter responses at the top layer $L$ over all the filters, positions and times. The spatial and temporal pooling reflects the fact that we assume the model is stationary in spatial and temporal domains. If the dynamic texture is non-stationary in the spatial or temporal domain, then  the top layer filters $F_k^{(L)}$ are fully connected in the spatial or temporal domain, e.g., ${\cal D}_L$ is $1 \times 1$. 

A simple but consequential property of the ReLU non-linearity is that $h(r) = \max(0, r) = 1(r>0) r$, where $1()$ is the indicator function, so that $1(r>0) = 1$ if $r>0$ and $0$ otherwise. As a result, the scoring function $f(\I; w)$ is piecewise linear \cite{montufar2014number}, and each linear piece is defined by the multiple layers of binary activation variables $\delta_{k, x, t}^{(l)}(\I; w) = 1\left([F_k^{(l)}*\I](x, t)>0\right)$, which tells us whether a local spatial-temporal pattern represented by the $k$-th filter at layer $l$,  $F_k^{(l)}$, is detected at position $x$ and time $t$. Let $\delta(\I; w) = \left(\delta_{k, x, t}^{(l)}(\I; w), \forall l, k, x, t \right)$ be the activation pattern of $\I$. Then $\delta(\I; w)$ divides the image space into a large number of pieces according to the value of $\delta(\I; w)$. On each piece of image space with fixed $\delta(\I; w)$, the scoring function $f(\I; w)$ is linear, i.e.,  
\begin{equation}
f(\I; w) = a_{w, \delta(\I; w)}  + \langle \I, B_{w, \delta(\I; w)}\rangle,
\end{equation}
 where both $a$ and $B$ are defined by $\delta(\I; w)$ and  $w$. In fact, $B = \partial f(\I; w)/\partial \I$, and can be computed by back-propagation, with $h'(r) = 1(r>0)$. The back-propagation process defines a top-down deconvolution process \cite{zeiler2011adaptive}, where the filters at multiple layers become the basis functions at those layers, and the activation variables at different layers in $\delta(\I; w)$ become the coefficients of the basis functions in the top-down deconvolution. 

$p(\I; w)$ in  (\ref{eq:ConvNet-FRAME})  is an energy-based model \cite{lecun2006tutorial, Ng2011}, whose energy function is a combination of the $\ell_2$ norm $\|\I\|^2$ that comes from the reference distribution $q(\I)$ and the piecewise linear scoring function $f(\I; w)$, i.e., 
\begin{equation}
\begin{aligned}
 {\cal E}(\I; w) &= -f(\I; w) + \frac{1}{2} \|\I\|^2\\
 & =\frac{1}{2}  \|\I\|^2 - \left(a_{w, \delta(\I; w)}  + \langle \I, B_{w, \delta(\I; w)}\rangle \right)\\
 &= \frac{1}{2} \|\I - B_{w, \delta(\I; w)}\|^2 + {\rm const}, 
\label{eq:ConvNet-FRAME3}
\end{aligned}
\end{equation}
where ${\rm const} = -a_{w, \delta(\I; w)} - \|B_{w, \delta(\I; w)}\|^2/2$, which is constant on the piece of image space with fixed $\delta(\I; w)$.

Since ${\cal E}(\I; w)$ is a piecewise quadratic function,  $p(\I; w)$ is piecewise Gaussian. On the piece of image space $\{\I: \delta(\I; w) = \delta\}$, where $\delta$ is a fixed value of $\delta(\I; w)$, $p(\I; w)$ is ${\rm N}(B_{w, \delta}, {\bf 1})$ truncated to  $\{\I: \delta(\I; w) = \delta\}$, where we use ${\bf 1}$ to denote the identity matrix. If the mean of this Gaussian piece,  $B_{w, \delta}$,  is within $\{\I: \delta(\I; w) = \delta\}$, then $B_{w, \delta}$ is also a local mode, and this local mode $\I$ satisfies a hierarchical auto-encoder, with a bottom-up encoding process $\delta = \delta(\I; w)$, and a top-down decoding process $\I = B_{w, \delta}$. In general, for an image sequence $\I$, $B_{w, \delta(\I; w)}$ can be considered a reconstruction of $\I$, and this reconstruction is exact if $\I$ is a local mode of ${\cal E}(\I; w)$. 

\subsection{Sampling and learning algorithm}

One can sample from $p(\I; w)$  of model (\ref{eq:ConvNet-FRAME}) by the Langevin dynamics: 
\begin{equation}
\I_{\tau+1} = \I_{\tau} - \frac{\epsilon^2}{2} \left[\I_\tau - \B_{w, \delta(\I_\tau; w)}\right] + \epsilon Z_\tau, \label{eq:Langevin}
\end{equation}
 where $\tau$ indexes the time steps,  $\epsilon$ is the step size, and  $Z_\tau \sim {\rm N}(0, {\bf 1})$. The dynamics is driven by the reconstruction error $\I -\B_{w, \delta(\I; w)}$. The finiteness of the step size $\epsilon$ can be corrected by a Metropolis-Hastings acceptance-rejection step. The Langevin dynamics can be extended to Hamiltonian Monte Carlo \cite{neal2011mcmc} or more sophisticated versions \cite{girolami2011riemann}.



The learning of $w$ from training image sequences $\{\I_m, m = 1, ..., M\}$ can be accomplished by the maximum likelihood. Let $L(w) = \sum_{m=1}^{M} \log p(\I; w)/M$, 
with $p(\I; w)$ defined in (\ref{eq:ConvNet-FRAME}), 
\begin{equation}
%\begin{aligned}
\frac{\partial L(w)}{\partial w} = \frac{1}{M} \sum_{m=1}^{M} \frac{\partial}{\partial w} f(\I_m; w) 
   -  \E_{w} \left[ \frac{\partial}{\partial w} f(\I; w) \right].
%\end{aligned}
\end{equation}
The expectation can be approximated by the Monte Carlo samples \cite{younes1999convergence} produced by the Langevin dynamics. See Algorithm \ref{code:FRAME}  for a description of the learning and sampling algorithm. The algorithm keeps synthesizing image sequences from the current model, and updating the model parameters in order to match the synthesized image sequences to the observed image sequences. The learning algorithm keeps shifting the probability density or low energy regions of the model from the synthesized data towards the observed data. 

 
\begin{algorithm}
\caption{Learning and sampling algorithm}
\label{code:FRAME}
\begin{algorithmic}[1]

\REQUIRE ~~\\
(1)  training image sequences $\{\I_m, m=1,...,M\}$ \\
(2) number of synthesized image sequences $\tilde{M}$\\
(3) number of Langevin steps $l$\\
(4) number of learning iterations $T$

\ENSURE~~\\
(1) estimated parameters $w$\\
(2) synthesized image sequences $\{\tI_m, m = 1, ..., \tilde{M}\}$ 

\item[]
\STATE Let $t\leftarrow 0$, initialize $w^{(0)}$.
\STATE Initialize $\tI_m$, for $m = 1, ..., \tilde{M}$. 
\REPEAT 
\STATE For each $m$, run $l$ steps of Langevin dynamics to update $\tI_m$, i.e., starting from the current $\tI_m$, each step 
follows equation (\ref{eq:Langevin}). 
\STATE Calculate  $H^{\obs} = \sum_{m=1}^{M} \frac{\partial}{\partial w} f(\I_m; w^{(t)})/M$, and
$H^{\syn} =  \sum_{m=1}^{\tM} \frac{\partial}{\partial w} f(\tI_m; w^{(t)})/\tM$.
\STATE Update $w^{(t+1)} \leftarrow w^{(t)} + \eta_t ( H^{\obs} - H^{\syn}) $,  with step size $\eta_t$. 
\STATE Let $t \leftarrow t+1$
\UNTIL $t = T$
\end{algorithmic}
\end{algorithm}

In the learning algorithm, the Langevin sampling step involves the computation of $\partial f(\I; w)/\partial \I$, and the parameter updating step involves the computation of $\partial f(\I; w)/\partial w$. Because of the ConvNet structure of $f(\I; w)$, both gradients can be computed efficiently by back-propagation, and the two gradients share most of their chain rule computations in back-propagation. 
%The whole learning and sampling algorithm can be considered an alternating back-propagation algorithm. 
In term of MCMC sampling, the Langevin dynamics samples from an evolving distribution because $w^{(t)}$ keeps changing. Thus the learning and sampling algorithm runs non-stationary chains. 

\subsection{Adversarial interpretation} \label{sect:adv}

Our model is an energy-based model 
\begin{align} 
p(\I; w) =\frac{1}{Z(w)}  \exp[-{\cal E}(\I; w)].
\end{align}
 The update of $w$ is based on $L'(w)$ which can be approximated by 
 \begin{align}
 \frac{1}{\tilde{M}} \sum_{m=1}^{\tilde{M}} \frac{\partial}{\partial w} {\cal E}(\tilde{\I}_m; w)
 - \frac{1}{M} \sum_{m=1}^{M} \frac{\partial}{\partial w} {\cal E}(\I_m; w), 
 \end{align}  
where $\{\tilde{\I}_m, m = 1, ..., \tilde{M}\}$ are the synthesized image sequences that are generated by the Langevin dynamics. At the zero temperature limit, the Langevin dynamics becomes gradient descent:
\begin{equation}
\tilde{\I}_{\tau+1} = \tilde{\I}_{\tau} - \frac{\epsilon^2}{2} \frac{\partial}{\partial \tilde{\I}} {\cal E}(\tilde{\I}_{\tau}; w). \label{eq:Langevin1}
\end{equation}
Consider the value function $V(\tilde{\I}_m, m = 1, ..., \tilde{M}; w)$: 
 \begin{align}
 \frac{1}{\tilde{M}} \sum_{m=1}^{\tilde{M}} {\cal E}(\tilde{\I}_m; w)
 -\frac{1}{M} \sum_{m=1}^{M}  {\cal E}(\I_m; w). 
 \end{align}  
The updating of $w$ is to increase $V$ by shifting the low energy regions from the synthesized image sequences $\{\tilde{\I}_m\}$ to the observed image sequences $\{\I_m\}$, whereas the updating of $\{\tilde{\I}_m, m = 1, ..., \tilde{M}\}$ is to decrease $V$ by moving the synthesized image sequences towards the low energy regions. This is an adversarial interpretation of the learning and sampling algorithm.
% but the value function is different from that of the generative adversarial learning  \cite{goodfellow2014generative}. 
 It can also be considered a generalization of the herding method \cite{welling2009herding} from exponential family models to general energy-based models. 

In our work, we let $-{\cal E}(\I; w) = f(\I; w) - \|\I\|^2/2\sigma^2$. We can also let $-{\cal E}(\I; w) = f(\I; w)$ by assuming a uniform reference distribution $q(\I)$. Our experiments show that the model with the uniform $q$ can also synthesize realistic dynamic patterns.  

The generative adversarial learning  \cite{goodfellow2014generative, vondrick2016generating} has a generator network. Unlike our model which is based on a bottom-up ConvNet $f(\I; w)$, the generator network generates $\I$ by a top-down ConvNet $\I = g(X; \tilde{w})$ where $X$ is a latent vector that follows a known prior distribution, and $\tilde{w}$ collects the parameters of the top-down ConvNet. Recently \cite{HanLu2016} developed an alternating back-propagation algorithm to train the generator network, without involving an extra network. More recently, \cite{xie2016cooperative} developed a cooperative training method that recruits a generator network $g(X; \tilde{w})$ to reconstruct and regenerate the synthesized image sequences $\{\tilde{\I}_m\}$ to speed up MCMC sampling. 

\section{Experiments}

We learn the spatial-temporal generative ConvNet from video clips collected from DynTex++ dataset of \cite{ghanem2010maximum} and the Internet.  The code in the experiments is based on the MatConvNet of \cite{matconvnn} and MexConv3D of ~\cite{MexConv3D}.

We show the synthesis results by displaying the frames in the video sequences. We have posted the synthesis results on the project page \url{http://www.stat.ucla.edu/~jxie/STGConvNet/STGConvNet.html}, so that the reader can watch the videos.


\subsection{Experiment 1: Generating dynamic textures with both spatial and temporal stationarity}


\begin{figure}
\begin{center}
%\includegraphics[width=.15\linewidth]{./dynamicTexture/water/observed_sequence_1/010.jpg}
%\includegraphics[width=.15\linewidth]{./dynamicTexture/water/observed_sequence_1/011.jpg}
%\includegraphics[width=.15\linewidth]{./dynamicTexture/water/observed_sequence_1/012.jpg}
%\includegraphics[width=.15\linewidth]{./dynamicTexture/water/observed_sequence_1/013.jpg}
%\includegraphics[width=.15\linewidth]{./dynamicTexture/water/observed_sequence_1/014.jpg}
%\includegraphics[width=.15\linewidth]{./dynamicTexture/water/observed_sequence_1/015.jpg} \\[3px]
%
%\includegraphics[width=.15\linewidth]{./dynamicTexture/water/synthesis_chain_1/image_0010.jpg}
%\includegraphics[width=.15\linewidth]{./dynamicTexture/water/synthesis_chain_1/image_0011.jpg}
%\includegraphics[width=.15\linewidth]{./dynamicTexture/water/synthesis_chain_1/image_0012.jpg}
%\includegraphics[width=.15\linewidth]{./dynamicTexture/water/synthesis_chain_1/image_0013.jpg}
%\includegraphics[width=.15\linewidth]{./dynamicTexture/water/synthesis_chain_1/image_0014.jpg}
%\includegraphics[width=.15\linewidth]{./dynamicTexture/water/synthesis_chain_1/image_0015.jpg}  \\ (a) ocean	\\[3px]	

%\includegraphics[width=.15\linewidth]{./dynamicTexture/water/training/010.jpg}
%\includegraphics[width=.15\linewidth]{./dynamicTexture/water/training/011.jpg}
%\includegraphics[width=.15\linewidth]{./dynamicTexture/water/training/012.jpg}
%\includegraphics[width=.15\linewidth]{./dynamicTexture/water/training/013.jpg}
%\includegraphics[width=.15\linewidth]{./dynamicTexture/water/training/014.jpg}
%\includegraphics[width=.15\linewidth]{./dynamicTexture/water/training/015.jpg} \\[3px]

%\includegraphics[width=.15\linewidth]{./dynamicTexture/water_layerBylayer/synthesis/image_0010.jpg}
%\includegraphics[width=.15\linewidth]{./dynamicTexture/water_layerBylayer/synthesis/image_0011.jpg}
%\includegraphics[width=.15\linewidth]{./dynamicTexture/water_layerBylayer/synthesis/image_0012.jpg}
%\includegraphics[width=.15\linewidth]{./dynamicTexture/water_layerBylayer/synthesis/image_0013.jpg}	
%\includegraphics[width=.15\linewidth]{./dynamicTexture/water_layerBylayer/synthesis/image_0014.jpg}
%\includegraphics[width=.15\linewidth]{./dynamicTexture/water_layerBylayer/synthesis/image_0015.jpg}  \\ (a) ocean	\\[3px]	

%\includegraphics[width=.15\linewidth]{./dynamicTexture/water2/observed_sequence_1/010.jpg}
%\includegraphics[width=.15\linewidth]{./dynamicTexture/water2/observed_sequence_1/011.jpg}
%\includegraphics[width=.15\linewidth]{./dynamicTexture/water2/observed_sequence_1/012.jpg}
%\includegraphics[width=.15\linewidth]{./dynamicTexture/water2/observed_sequence_1/013.jpg}
%\includegraphics[width=.15\linewidth]{./dynamicTexture/water2/observed_sequence_1/014.jpg}
%\includegraphics[width=.15\linewidth]{./dynamicTexture/water2/observed_sequence_1/015.jpg}\\[3px]
%
%\includegraphics[width=.15\linewidth]{./dynamicTexture/water2/synthesis_chain_1/image_0010.jpg}
%\includegraphics[width=.15\linewidth]{./dynamicTexture/water2/synthesis_chain_1/image_0011.jpg}
%\includegraphics[width=.15\linewidth]{./dynamicTexture/water2/synthesis_chain_1/image_0012.jpg}
%\includegraphics[width=.15\linewidth]{./dynamicTexture/water2/synthesis_chain_1/image_0013.jpg}
%\includegraphics[width=.15\linewidth]{./dynamicTexture/water2/synthesis_chain_1/image_0014.jpg}
%\includegraphics[width=.15\linewidth]{./dynamicTexture/water2/synthesis_chain_1/image_0015.jpg}  \\ (a) river	\\[3px]	

\includegraphics[width=.13\linewidth]{./dynamicTexture/water4/observed_sequence_1/010.jpg}
\includegraphics[width=.13\linewidth]{./dynamicTexture/water4/observed_sequence_1/011.jpg}
\includegraphics[width=.13\linewidth]{./dynamicTexture/water4/observed_sequence_1/012.jpg}
\includegraphics[width=.13\linewidth]{./dynamicTexture/water4/observed_sequence_1/013.jpg}
\includegraphics[width=.13\linewidth]{./dynamicTexture/water4/observed_sequence_1/014.jpg}
\includegraphics[width=.13\linewidth]{./dynamicTexture/water4/observed_sequence_1/015.jpg}
\includegraphics[width=.13\linewidth]{./dynamicTexture/water4/observed_sequence_1/016.jpg}\\[3px]

\includegraphics[width=.13\linewidth]{./dynamicTexture/water4/synthesis_chain_2/image_0010.jpg}
\includegraphics[width=.13\linewidth]{./dynamicTexture/water4/synthesis_chain_2/image_0011.jpg}
\includegraphics[width=.13\linewidth]{./dynamicTexture/water4/synthesis_chain_2/image_0012.jpg}
\includegraphics[width=.13\linewidth]{./dynamicTexture/water4/synthesis_chain_2/image_0013.jpg}
\includegraphics[width=.13\linewidth]{./dynamicTexture/water4/synthesis_chain_2/image_0014.jpg}
\includegraphics[width=.13\linewidth]{./dynamicTexture/water4/synthesis_chain_2/image_0015.jpg}
\includegraphics[width=.13\linewidth]{./dynamicTexture/water4/synthesis_chain_2/image_0016.jpg} \\[3px]	

\includegraphics[width=.13\linewidth]{./dynamicTexture/water4/synthesis_chain_1/image_0010.jpg}
\includegraphics[width=.13\linewidth]{./dynamicTexture/water4/synthesis_chain_1/image_0011.jpg}
\includegraphics[width=.13\linewidth]{./dynamicTexture/water4/synthesis_chain_1/image_0012.jpg}
\includegraphics[width=.13\linewidth]{./dynamicTexture/water4/synthesis_chain_1/image_0013.jpg}
\includegraphics[width=.13\linewidth]{./dynamicTexture/water4/synthesis_chain_1/image_0014.jpg}
\includegraphics[width=.13\linewidth]{./dynamicTexture/water4/synthesis_chain_1/image_0015.jpg}
\includegraphics[width=.13\linewidth]{./dynamicTexture/water4/synthesis_chain_1/image_0016.jpg}  \\ (a) river	\\[3px]	


\includegraphics[width=.13\linewidth]{./dynamicTexture/ocean/observed_sequence_1/010.jpg}
\includegraphics[width=.13\linewidth]{./dynamicTexture/ocean/observed_sequence_1/011.jpg}
\includegraphics[width=.13\linewidth]{./dynamicTexture/ocean/observed_sequence_1/012.jpg}
\includegraphics[width=.13\linewidth]{./dynamicTexture/ocean/observed_sequence_1/013.jpg}
\includegraphics[width=.13\linewidth]{./dynamicTexture/ocean/observed_sequence_1/014.jpg}
\includegraphics[width=.13\linewidth]{./dynamicTexture/ocean/observed_sequence_1/015.jpg}
\includegraphics[width=.13\linewidth]{./dynamicTexture/ocean/observed_sequence_1/016.jpg}\\
[3px]

\includegraphics[width=.13\linewidth]{./dynamicTexture/ocean/synthesis_chain_2/image_0010.jpg}
\includegraphics[width=.13\linewidth]{./dynamicTexture/ocean/synthesis_chain_2/image_0011.jpg}
\includegraphics[width=.13\linewidth]{./dynamicTexture/ocean/synthesis_chain_2/image_0012.jpg}
\includegraphics[width=.13\linewidth]{./dynamicTexture/ocean/synthesis_chain_2/image_0013.jpg}
\includegraphics[width=.13\linewidth]{./dynamicTexture/ocean/synthesis_chain_2/image_0014.jpg}
\includegraphics[width=.13\linewidth]{./dynamicTexture/ocean/synthesis_chain_2/image_0015.jpg}
\includegraphics[width=.13\linewidth]{./dynamicTexture/ocean/synthesis_chain_2/image_0016.jpg}\\[3px]

\includegraphics[width=.13\linewidth]{./dynamicTexture/ocean/synthesis_chain_1/image_0010.jpg}
\includegraphics[width=.13\linewidth]{./dynamicTexture/ocean/synthesis_chain_1/image_0011.jpg}
\includegraphics[width=.13\linewidth]{./dynamicTexture/ocean/synthesis_chain_1/image_0012.jpg}
\includegraphics[width=.13\linewidth]{./dynamicTexture/ocean/synthesis_chain_1/image_0013.jpg}
\includegraphics[width=.13\linewidth]{./dynamicTexture/ocean/synthesis_chain_1/image_0014.jpg}
\includegraphics[width=.13\linewidth]{./dynamicTexture/ocean/synthesis_chain_1/image_0015.jpg}
\includegraphics[width=.13\linewidth]{./dynamicTexture/ocean/synthesis_chain_1/image_0016.jpg}  \\ (b) ocean	\\[3px]	
%\includegraphics[width=.15\linewidth]{./dynamicTexture/water2_layerBylayer/synthesis/image_0010.jpg}
%\includegraphics[width=.15\linewidth]{./dynamicTexture/water2_layerBylayer/synthesis/image_0011.jpg}
%\includegraphics[width=.15\linewidth]{./dynamicTexture/water2_layerBylayer/synthesis/image_0012.jpg}
%\includegraphics[width=.15\linewidth]{./dynamicTexture/water2_layerBylayer/synthesis/image_0013.jpg}	
%\includegraphics[width=.15\linewidth]{./dynamicTexture/water2_layerBylayer/synthesis/image_0014.jpg}
%\includegraphics[width=.15\linewidth]{./dynamicTexture/water2_layerBylayer/synthesis/image_0015.jpg} \\ (b) river	\\[3px]	
	\caption{Synthesizing dynamic textures with both spatial and temporal stationarity. For each category, the first row displays the frames of the observed sequence, and the second and third  rows  display the corresponding frames of two synthesized sequences generated by the learning algorithm. (a) river. (b) ocean. }
	\label{fig:DTresults1}
\end{center}
\end{figure}

\begin{figure}
\begin{center}
%\includegraphics[width=.15\linewidth]{./dynamicTexture/fallingwater1/training/010.jpg}
%\includegraphics[width=.15\linewidth]{./dynamicTexture/fallingwater1/training/011.jpg}
%\includegraphics[width=.15\linewidth]{./dynamicTexture/fallingwater1/training/012.jpg}
%\includegraphics[width=.15\linewidth]{./dynamicTexture/fallingwater1/training/013.jpg}
%\includegraphics[width=.15\linewidth]{./dynamicTexture/fallingwater1/training/014.jpg}
%\includegraphics[width=.15\linewidth]{./dynamicTexture/fallingwater1/training/015.jpg} \\[3px]	
%
%\includegraphics[width=.15\linewidth]{./dynamicTexture/fallingwater1/synthesis/image_0010.jpg}
%\includegraphics[width=.15\linewidth]{./dynamicTexture/fallingwater1/synthesis/image_0011.jpg}
%\includegraphics[width=.15\linewidth]{./dynamicTexture/fallingwater1/synthesis/image_0012.jpg}
%\includegraphics[width=.15\linewidth]{./dynamicTexture/fallingwater1/synthesis/image_0013.jpg}	
%\includegraphics[width=.15\linewidth]{./dynamicTexture/fallingwater1/synthesis/image_0014.jpg}
%\includegraphics[width=.15\linewidth]{./dynamicTexture/fallingwater1/synthesis/image_0015.jpg}	\\[3px]
%\includegraphics[width=.13\linewidth]{./dynamicTexture/fallingwater2/training1/010.jpg}
%\includegraphics[width=.13\linewidth]{./dynamicTexture/fallingwater2/training1/011.jpg}
%\includegraphics[width=.13\linewidth]{./dynamicTexture/fallingwater2/training1/012.jpg}
%\includegraphics[width=.13\linewidth]{./dynamicTexture/fallingwater2/training1/013.jpg}
%\includegraphics[width=.13\linewidth]{./dynamicTexture/fallingwater2/training1/014.jpg} 
%\includegraphics[width=.13\linewidth]{./dynamicTexture/fallingwater2/training1/015.jpg}
%\includegraphics[width=.13\linewidth]{./dynamicTexture/fallingwater2/training1/016.jpg}
%\\[3px]	
%\includegraphics[width=.13\linewidth]{./dynamicTexture/fallingwater2/synthesis/image_0010.jpg}
%\includegraphics[width=.13\linewidth]{./dynamicTexture/fallingwater2/synthesis/image_0011.jpg}
%\includegraphics[width=.13\linewidth]{./dynamicTexture/fallingwater2/synthesis/image_0012.jpg}
%\includegraphics[width=.13\linewidth]{./dynamicTexture/fallingwater2/synthesis/image_0013.jpg}	
%\includegraphics[width=.13\linewidth]{./dynamicTexture/fallingwater2/synthesis/image_0014.jpg}
%\includegraphics[width=.13\linewidth]{./dynamicTexture/fallingwater2/synthesis/image_0015.jpg}	
%\includegraphics[width=.13\linewidth]{./dynamicTexture/fallingwater2/synthesis/image_0016.jpg}	
%\\	
%(a) spring water \\[3px]  

%\includegraphics[width=.15\linewidth]{./dynamicTexture/flashing_light/observed_sequence_1/019.jpg}
\includegraphics[width=.15\linewidth]{./dynamicTexture/flashing_light/observed_sequence_1/020.jpg}
\includegraphics[width=.15\linewidth]{./dynamicTexture/flashing_light/observed_sequence_1/021.jpg}
\includegraphics[width=.15\linewidth]{./dynamicTexture/flashing_light/observed_sequence_1/022.jpg}
\includegraphics[width=.15\linewidth]{./dynamicTexture/flashing_light/observed_sequence_1/023.jpg} 
\includegraphics[width=.15\linewidth]{./dynamicTexture/flashing_light/observed_sequence_1/024.jpg} 
\includegraphics[width=.15\linewidth]{./dynamicTexture/flashing_light/observed_sequence_1/025.jpg}\\[3px]	

%\includegraphics[width=.15\linewidth]{./dynamicTexture/flashing_light/synthesis_chain_3/image_0019.jpg}
%%\includegraphics[width=.15\linewidth]{./dynamicTexture/flashing_light/synthesis_chain_3/image_0020.jpg}
%%\includegraphics[width=.15\linewidth]{./dynamicTexture/flashing_light/synthesis_chain_3/image_0021.jpg}
%%\includegraphics[width=.15\linewidth]{./dynamicTexture/flashing_light/synthesis_chain_3/image_0022.jpg}
%%\includegraphics[width=.15\linewidth]{./dynamicTexture/flashing_light/synthesis_chain_3/image_0023.jpg} 
%%\includegraphics[width=.15\linewidth]{./dynamicTexture/flashing_light/synthesis_chain_3/image_0024.jpg} 
%%\includegraphics[width=.15\linewidth]{./dynamicTexture/flashing_light/synthesis_chain_3/image_0025.jpg}\\[3px]	

%\includegraphics[width=.15\linewidth]{./dynamicTexture/flashing_light/synthesis_chain_2/image_0019.jpg}
\includegraphics[width=.15\linewidth]{./dynamicTexture/flashing_light/synthesis_chain_2/image_0020.jpg}
\includegraphics[width=.15\linewidth]{./dynamicTexture/flashing_light/synthesis_chain_2/image_0021.jpg}
\includegraphics[width=.15\linewidth]{./dynamicTexture/flashing_light/synthesis_chain_2/image_0022.jpg}
\includegraphics[width=.15\linewidth]{./dynamicTexture/flashing_light/synthesis_chain_2/image_0023.jpg} 
\includegraphics[width=.15\linewidth]{./dynamicTexture/flashing_light/synthesis_chain_2/image_0024.jpg} 
\includegraphics[width=.15\linewidth]{./dynamicTexture/flashing_light/synthesis_chain_2/image_0025.jpg}\\ (a) flashing lights \\[3px]  	

%\includegraphics[width=.13\linewidth]{./dynamicTexture/fire_pot/synthesis_chain_3/image_0035.jpg}
%\includegraphics[width=.13\linewidth]{./dynamicTexture/fire_pot/synthesis_chain_3/image_0036.jpg}
%\includegraphics[width=.13\linewidth]{./dynamicTexture/fire_pot/synthesis_chain_3/image_0037.jpg}
%\includegraphics[width=.13\linewidth]{./dynamicTexture/fire_pot/synthesis_chain_3/image_0038.jpg}	
%\includegraphics[width=.13\linewidth]{./dynamicTexture/fire_pot/synthesis_chain_3/image_0039.jpg}
%\includegraphics[width=.13\linewidth]{./dynamicTexture/fire_pot/synthesis_chain_3/image_0040.jpg}
%\includegraphics[width=.13\linewidth]{./dynamicTexture/fire_pot/synthesis_chain_3/image_0041.jpg}\\	[3px]	
%
%\includegraphics[width=.13\linewidth]{./dynamicTexture/fire_pot/synthesis_chain_1/image_0035.jpg}
%\includegraphics[width=.13\linewidth]{./dynamicTexture/fire_pot/synthesis_chain_1/image_0036.jpg}
%\includegraphics[width=.13\linewidth]{./dynamicTexture/fire_pot/synthesis_chain_1/image_0037.jpg}
%\includegraphics[width=.13\linewidth]{./dynamicTexture/fire_pot/synthesis_chain_1/image_0038.jpg}	
%\includegraphics[width=.13\linewidth]{./dynamicTexture/fire_pot/synthesis_chain_1/image_0039.jpg}
%\includegraphics[width=.13\linewidth]{./dynamicTexture/fire_pot/synthesis_chain_1/image_0040.jpg}
%\includegraphics[width=.13\linewidth]{./dynamicTexture/fire_pot/synthesis_chain_1/image_0041.jpg}\\	



\includegraphics[width=.15\linewidth]{./dynamicTexture/water_fountain/observed_sequence_1/019.jpg}
\includegraphics[width=.15\linewidth]{./dynamicTexture/water_fountain/observed_sequence_1/020.jpg}
\includegraphics[width=.15\linewidth]{./dynamicTexture/water_fountain/observed_sequence_1/021.jpg}
\includegraphics[width=.15\linewidth]{./dynamicTexture/water_fountain/observed_sequence_1/022.jpg}
\includegraphics[width=.15\linewidth]{./dynamicTexture/water_fountain/observed_sequence_1/023.jpg} 
\includegraphics[width=.15\linewidth]{./dynamicTexture/water_fountain/observed_sequence_1/024.jpg} \\[3px]	
%\includegraphics[width=.15\linewidth]{./dynamicTexture/water_fountain/observed_sequence_1/025.jpg}\\[3px]	

%%\includegraphics[width=.15\linewidth]{./dynamicTexture/water_fountain/synthesis_chain_3/image_0019.jpg}
%%\includegraphics[width=.15\linewidth]{./dynamicTexture/water_fountain/synthesis_chain_3/image_0020.jpg}
%%\includegraphics[width=.15\linewidth]{./dynamicTexture/water_fountain/synthesis_chain_3/image_0021.jpg}
%%\includegraphics[width=.15\linewidth]{./dynamicTexture/water_fountain/synthesis_chain_3/image_0022.jpg}
%%\includegraphics[width=.15\linewidth]{./dynamicTexture/water_fountain/synthesis_chain_3/image_0023.jpg} 
%%\includegraphics[width=.15\linewidth]{./dynamicTexture/water_fountain/synthesis_chain_3/image_0024.jpg} \\[3px]	
%\includegraphics[width=.15\linewidth]{./dynamicTexture/water_fountain/synthesis_chain_3/image_0025.jpg}\\[3px]	

\includegraphics[width=.15\linewidth]{./dynamicTexture/water_fountain/synthesis_chain_2/image_0019.jpg}
\includegraphics[width=.15\linewidth]{./dynamicTexture/water_fountain/synthesis_chain_2/image_0020.jpg}
\includegraphics[width=.15\linewidth]{./dynamicTexture/water_fountain/synthesis_chain_2/image_0021.jpg}
\includegraphics[width=.15\linewidth]{./dynamicTexture/water_fountain/synthesis_chain_2/image_0022.jpg}
\includegraphics[width=.15\linewidth]{./dynamicTexture/water_fountain/synthesis_chain_2/image_0023.jpg} 
\includegraphics[width=.15\linewidth]{./dynamicTexture/water_fountain/synthesis_chain_2/image_0024.jpg} \\ (b) fountain\\[3px] 
%\includegraphics[width=.15\linewidth]{./dynamicTexture/water_fountain/synthesis_chain_2/image_0025.jpg}\\ (b) fountain\\[3px] 



\includegraphics[width=.15\linewidth]{./dynamicTexture/fire_pot/observed_sequence_1/035.jpg}
\includegraphics[width=.15\linewidth]{./dynamicTexture/fire_pot/observed_sequence_1/036.jpg}
\includegraphics[width=.15\linewidth]{./dynamicTexture/fire_pot/observed_sequence_1/037.jpg}
\includegraphics[width=.15\linewidth]{./dynamicTexture/fire_pot/observed_sequence_1/038.jpg}
\includegraphics[width=.15\linewidth]{./dynamicTexture/fire_pot/observed_sequence_1/039.jpg} 
\includegraphics[width=.15\linewidth]{./dynamicTexture/fire_pot/observed_sequence_1/040.jpg}\\[3px]	 
%\includegraphics[width=.13\linewidth]{./dynamicTexture/fire_pot/observed_sequence_1/041.jpg}\\[3px]	

%%\includegraphics[width=.15\linewidth]{./dynamicTexture/fire_pot/synthesis_chain_3/image_0035.jpg}
%%\includegraphics[width=.15\linewidth]{./dynamicTexture/fire_pot/synthesis_chain_3/image_0036.jpg}
%%\includegraphics[width=.15\linewidth]{./dynamicTexture/fire_pot/synthesis_chain_3/image_0037.jpg}
%%\includegraphics[width=.15\linewidth]{./dynamicTexture/fire_pot/synthesis_chain_3/image_0038.jpg}	
%%\includegraphics[width=.15\linewidth]{./dynamicTexture/fire_pot/synthesis_chain_3/image_0039.jpg}
%%\includegraphics[width=.15\linewidth]{./dynamicTexture/fire_pot/synthesis_chain_3/image_0040.jpg}\\	[3px]	
%\includegraphics[width=.13\linewidth]{./dynamicTexture/fire_pot/synthesis_chain_3/image_0041.jpg}\\	[3px]	

\includegraphics[width=.15\linewidth]{./dynamicTexture/fire_pot/synthesis_chain_1/image_0035.jpg}
\includegraphics[width=.15\linewidth]{./dynamicTexture/fire_pot/synthesis_chain_1/image_0036.jpg}
\includegraphics[width=.15\linewidth]{./dynamicTexture/fire_pot/synthesis_chain_1/image_0037.jpg}
\includegraphics[width=.15\linewidth]{./dynamicTexture/fire_pot/synthesis_chain_1/image_0038.jpg}	
\includegraphics[width=.15\linewidth]{./dynamicTexture/fire_pot/synthesis_chain_1/image_0039.jpg}
\includegraphics[width=.15\linewidth]{./dynamicTexture/fire_pot/synthesis_chain_1/image_0040.jpg}\\
%\includegraphics[width=.15\linewidth]{./dynamicTexture/fire_pot/synthesis_chain_1/image_0041.jpg}\\	
(c) burning fire heating a pot \\[3px]  

%%\includegraphics[width=.15\linewidth]{./dynamicTexture/washing_machine/observed_sequence_1/030.jpg}
%\includegraphics[width=.15\linewidth]{./dynamicTexture/washing_machine/observed_sequence_1/031.jpg}
%\includegraphics[width=.15\linewidth]{./dynamicTexture/washing_machine/observed_sequence_1/032.jpg}
%\includegraphics[width=.15\linewidth]{./dynamicTexture/washing_machine/observed_sequence_1/033.jpg}
%\includegraphics[width=.15\linewidth]{./dynamicTexture/washing_machine/observed_sequence_1/034.jpg}
%\includegraphics[width=.15\linewidth]{./dynamicTexture/washing_machine/observed_sequence_1/035.jpg}
%\includegraphics[width=.15\linewidth]{./dynamicTexture/washing_machine/observed_sequence_1/036.jpg} \\[3px]	
%
%%\includegraphics[width=.15\linewidth]{./dynamicTexture/washing_machine/synthesis_chain_1/image_0030.jpg}
%\includegraphics[width=.15\linewidth]{./dynamicTexture/washing_machine/synthesis_chain_1/image_0031.jpg}
%\includegraphics[width=.15\linewidth]{./dynamicTexture/washing_machine/synthesis_chain_1/image_0032.jpg}
%\includegraphics[width=.15\linewidth]{./dynamicTexture/washing_machine/synthesis_chain_1/image_0033.jpg}	
%\includegraphics[width=.15\linewidth]{./dynamicTexture/washing_machine/synthesis_chain_1/image_0034.jpg}
%\includegraphics[width=.15\linewidth]{./dynamicTexture/washing_machine/synthesis_chain_1/image_0035.jpg}
%\includegraphics[width=.15\linewidth]{./dynamicTexture/washing_machine/synthesis_chain_1/image_0036.jpg}	\\[3px]  
%
%
%%\includegraphics[width=.15\linewidth]{./dynamicTexture/washing_machine/synthesis_chain_3/image_0030.jpg}
%\includegraphics[width=.15\linewidth]{./dynamicTexture/washing_machine/synthesis_chain_3/image_0031.jpg}
%\includegraphics[width=.15\linewidth]{./dynamicTexture/washing_machine/synthesis_chain_3/image_0032.jpg}
%\includegraphics[width=.15\linewidth]{./dynamicTexture/washing_machine/synthesis_chain_3/image_0033.jpg}	
%\includegraphics[width=.15\linewidth]{./dynamicTexture/washing_machine/synthesis_chain_3/image_0034.jpg}
%\includegraphics[width=.15\linewidth]{./dynamicTexture/washing_machine/synthesis_chain_3/image_0035.jpg}	
%\includegraphics[width=.15\linewidth]{./dynamicTexture/washing_machine/synthesis_chain_3/image_0036.jpg}\\
%(d) washing machine \\[3px]


\includegraphics[width=.15\linewidth]{./dynamicTexture/water_stone2/observed_sequence_1/031.jpg}
\includegraphics[width=.15\linewidth]{./dynamicTexture/water_stone2/observed_sequence_1/032.jpg}
\includegraphics[width=.15\linewidth]{./dynamicTexture/water_stone2/observed_sequence_1/033.jpg}
\includegraphics[width=.15\linewidth]{./dynamicTexture/water_stone2/observed_sequence_1/034.jpg}
\includegraphics[width=.15\linewidth]{./dynamicTexture/water_stone2/observed_sequence_1/035.jpg}
\includegraphics[width=.15\linewidth]{./dynamicTexture/water_stone2/observed_sequence_1/036.jpg} \\[3px]	

%%\includegraphics[width=.15\linewidth]{./dynamicTexture/water_stone2/synthesis_chain_1/image_0031.jpg}
%%\includegraphics[width=.15\linewidth]{./dynamicTexture/water_stone2/synthesis_chain_1/image_0032.jpg}
%%\includegraphics[width=.15\linewidth]{./dynamicTexture/water_stone2/synthesis_chain_1/image_0033.jpg}	
%%\includegraphics[width=.15\linewidth]{./dynamicTexture/water_stone2/synthesis_chain_1/image_0034.jpg}
%%\includegraphics[width=.15\linewidth]{./dynamicTexture/water_stone2/synthesis_chain_1/image_0035.jpg}
%%\includegraphics[width=.15\linewidth]{./dynamicTexture/water_stone2/synthesis_chain_1/image_0036.jpg}	\\[3px]  

\includegraphics[width=.15\linewidth]{./dynamicTexture/water_stone2/synthesis_chain_2/image_0031.jpg}
\includegraphics[width=.15\linewidth]{./dynamicTexture/water_stone2/synthesis_chain_2/image_0032.jpg}
\includegraphics[width=.15\linewidth]{./dynamicTexture/water_stone2/synthesis_chain_2/image_0033.jpg}	
\includegraphics[width=.15\linewidth]{./dynamicTexture/water_stone2/synthesis_chain_2/image_0034.jpg}
\includegraphics[width=.15\linewidth]{./dynamicTexture/water_stone2/synthesis_chain_2/image_0035.jpg}
\includegraphics[width=.15\linewidth]{./dynamicTexture/water_stone2/synthesis_chain_2/image_0036.jpg}\\(d) spring water \\[3px]
	\caption{Synthesizing dynamic textures with only temporal stationarity. For each category, the first row displays the frames of the observed sequence, and the second row displays the corresponding frames of a synthesized sequence generated by the learning algorithm. (a) flashing lights. (b) fountain. (c) burning fire heating a pot. (d) spring water.}
	\label{fig:DTresults}
\end{center}
\end{figure}



\begin{figure}
\begin{center}
\includegraphics[width=.15\linewidth]{./dynamicTexture/fallingwater1/training/010.jpg}
\includegraphics[width=.15\linewidth]{./dynamicTexture/fallingwater1/training/011.jpg}
\includegraphics[width=.15\linewidth]{./dynamicTexture/fallingwater1/training/012.jpg}
\includegraphics[width=.15\linewidth]{./dynamicTexture/fallingwater1/training/013.jpg}
\includegraphics[width=.15\linewidth]{./dynamicTexture/fallingwater1/training/014.jpg}
\includegraphics[width=.15\linewidth]{./dynamicTexture/fallingwater1/training/015.jpg} \\[3px]	

\includegraphics[width=.15\linewidth]{./dynamicTexture/fallingwater1/synthesis/image_0010.jpg}
\includegraphics[width=.15\linewidth]{./dynamicTexture/fallingwater1/synthesis/image_0011.jpg}
\includegraphics[width=.15\linewidth]{./dynamicTexture/fallingwater1/synthesis/image_0012.jpg}
\includegraphics[width=.15\linewidth]{./dynamicTexture/fallingwater1/synthesis/image_0013.jpg}	
\includegraphics[width=.15\linewidth]{./dynamicTexture/fallingwater1/synthesis/image_0014.jpg}
\includegraphics[width=.15\linewidth]{./dynamicTexture/fallingwater1/synthesis/image_0015.jpg}	\\[3px]


\includegraphics[width=.15\linewidth]{./dt_comparison/water_fall50/synthesis_image_10.png}
\includegraphics[width=.15\linewidth]{./dt_comparison/water_fall50/synthesis_image_11.png}
\includegraphics[width=.15\linewidth]{./dt_comparison/water_fall50/synthesis_image_12.png}
\includegraphics[width=.15\linewidth]{./dt_comparison/water_fall50/synthesis_image_13.png}
\includegraphics[width=.15\linewidth]{./dt_comparison/water_fall50/synthesis_image_14.png}
\includegraphics[width=.15\linewidth]{./dt_comparison/water_fall50/synthesis_image_15.png} \\[3px]
%
%\includegraphics[width=.15\linewidth]{./dt_comparison/water_fall50_10_0.05_wtslow/synthesis_image_10.png}
%\includegraphics[width=.15\linewidth]{./dt_comparison/water_fall50_10_0.05_wtslow/synthesis_image_11.png}
%\includegraphics[width=.15\linewidth]{./dt_comparison/water_fall50_10_0.05_wtslow/synthesis_image_12.png}
%\includegraphics[width=.15\linewidth]{./dt_comparison/water_fall50_10_0.05_wtslow/synthesis_image_13.png}
%\includegraphics[width=.15\linewidth]{./dt_comparison/water_fall50_10_0.05_wtslow/synthesis_image_14.png}
%\includegraphics[width=.15\linewidth]{./dt_comparison/water_fall50_10_0.05_wtslow/synthesis_image_15.png}
	\caption{Comparison on synthesizing dynamic texture of waterfall. From top to bottom: segments of the observed sequence, synthesized sequence by
our method, and synthesized sequence by the
method of \cite{doretto2003dynamic}.}
	\label{fig:DT_comp}
\end{center}
\end{figure}


We first learn the model from dynamic textures that are stationary in both spatial and temporal domains. We use spatial-temporal filters that are convolutional in both spatial and temporal domains. The first layer has 120 $15 \times 15 \times 15$ filters with sub-sampling size of 7 pixels and frames. The second layer has 40 $7 \times 7 \times 7$ filters with sub-sampling size of 3.  The third layer has 20 $3 \times 3 \times 2$ filters with sub-sampling size of $2 \times 2 \times 1$. Figure \ref{fig:DTresults1} displays 2 results. For each category, the first row displays 7 frames of the observed sequence, while the second and third rows  show the corresponding frames of two synthesized sequences generated by the learning algorithm. 

We use the layer-by-layer learning scheme. Starting from the first layer, we sequentially add the layers one by one. Each time we learn the model and generate the synthesized image sequence using Algorithm \ref{code:FRAME}. While learning the new layer of filters, we refine the lower layers of filters with back-propagation.    

%Due to the limitation of the GPU memory and for the sake of computational efficiency, we use $\tilde{M}=3$ chain for Langevin sampling. Even with a single chain, the statistics can still be estimated because of stationarity in the temporal and spatial domains. 

We learn a spatial-temporal generative ConvNet for each category from one observed video that is prepared to be of the size 224 $\times$ 224 $\times$ 50 or 70. The range of intensities is  [0, 255]. Mean subtraction is used as pre-processing. We use $\tilde{M}=3$ chain for Langevin sampling. The number of Langevin iterations between every two consecutive updates of parameters, $l = 20$. The number of learning iterations $T = 1200$, where we add one more layer every $400$ iterations.  We use layer-specific learning rates, where the learning rate at the higher layer is less than that at the lower layer, in order to obtain stable convergence.  


\subsection{Experiment 2: Generating dynamic textures with only temporal stationarity}

Many dynamic textures have structured background and objects that are not stationary in the spatial domain. In this case, the network used in Experiment 1 may fail. However, we can modify the  network in Experiment 1  by using  filters that are fully connected in the spatial domain at the second layer. Specifically, the first layer has 120 $7 \times 7 \times 7$ filters with sub-sampling size of 3 pixels and frames. The second layer is a spatially fully connected layer, which contains 30 filters that are fully connected in the spatial domain but convolutional in the temporal domain. The temporal size of the filters is 4 frames with sub-sampling size of 2 frames in the temporal dimension.  Due to the spatial full connectivity at the second layer, the spatial domain of the feature maps at the third layer is reduced to $1 \times 1$.  The third layer has 5 $1 \times 1 \times 2$ filters with sub-sampling size of 1 in the temporal dimension. 

We  use  end-to-end learning scheme to learn the above 3-layer spatial-temporal generative ConvNet for dynamic textures. At each iteration, the 3 layers of filters are updated with 3 different layer-specific learning rates. The learning rate at the higher layer is much less than that at the lower layer to avoid the issue of large gradients. 

We learn a spatial-temporal generative ConvNet for each category from one training video. We synthesize $\tilde{M}=3$ videos using the Langevin dynamics. Figure \ref{fig:DTresults} displays the results. For each category, the first row shows 6 frames of the observed sequence (224 $\times$ 224 $\times$ 70), and the second  row shows the corresponding frames of a synthesized sequence generated by the learning algorithm. We use the same set of parameters for all the categories without tuning. Figure \ref{fig:DT_comp} compares our method to that of \cite{doretto2003dynamic}, which is a linear dynamic system model. The image sequence generated by this model appears more blurred than the sequence generated by our method. 


%We compare our method with FFT-LDS \cite{abraham2005dynamic}, GPDM \cite{wang2008gaussian}, HOSVD \cite{costantini2008higher}, and Doretto et al \cite{doretto2003dynamic}, in terms of PSNR and structural similarity measures
%(SSIM) on 6 dynamic texture videos. Table \ref{compExp}
%shows the average performances over the 6 videos.

%\begin{table}[h]
%\caption{Comparison of different dynamic texture models}\label{compExp}
%\vskip 0.02in
%\begin{center}
%\begin{footnotesize}
%\vskip 0.08in
%\begin{tabular}{|c|c|c|c|c|c|}
%\hline
%       & ours    & \cite{abraham2005dynamic}     & \cite{wang2008gaussian} & \cite{costantini2008higher} & \cite{doretto2003dynamic}\\ \hline \hline
%SSIM   & \textbf{0.6490}  & 0.1749  & 0.1134 &  0.2863 & 0.2122\\ \hline
%PSNR   & \textbf{19.2985}  & 10.5068  & 8.8770 & 13.4572 & 11.0300\\ \hline
%\end{tabular}
%\end{footnotesize}
%\end{center}
%\end{table}

%\subsection{Experiment 3: Generating sounds with temporal stationarity}
%
%We learn a single layer temporal generative ConvNet for sound patterns. This layer has 1000 $1 \times 2000$ filters with sub-sampling size of 500. ${\tilde M} = 3$ parallel chains for Langevin sampling are used. The number of Langevin iterations  $L=10$. The number of learning iterations $T=2000$. We collect some sound data from the Internet, which include animal sounds, natural sounds, and musical sounds. Each observed sound signal is a 5 second long clip, and is read into a $1 \times 60000$ dimensional vector with the sampling rate of 11025 Hertz, and the values are normalized into the range of $[-1.0, 1.0]$. Mean subtraction is used before training. Figure \ref{fig:sounds} shows 4 results by displaying the waveforms (1D plots) of the observed sound signals and the synthesized sound signals. For each category, the first plot is the observed sound, and the rest are 3 synthesized sounds generated by the learned  model. 

The learning of our model can be scaled up. We learn the fire pattern from 30 training videos, with mini-batch implementation. The size of each mini-batch is 10 videos. Each video contains 30 frames ($100 \times 100$ pixels). For each mini-batch, ${\tilde M} = 13$ parallel chains for Langevin sampling is used. For this experiment, we slightly modify the network by using $120$ $11 \times 11 \times 9$ filters with sub-sampling size of $5$ pixels and $4$ frames at the first layer, and 30 spatially fully connected filters with temporal size of 5 frames and sub-sampling size of $2$ at the second layer, while keeping the setting of the third layer unchanged. The number of learning iterations $T=1300$. Figure \ref{fig:flames} shows one frame for each of 30 observed sequences and the corresponding frame of the synthesized sequences. Two examples of synthesized sequences are also displayed.



\begin{figure}
\begin{center}
\includegraphics[width=.09\linewidth]{./big_data/observed_sequence/sequence_1/021.jpg}
\includegraphics[width=.09\linewidth]{./big_data/observed_sequence/sequence_2/021.jpg}
\includegraphics[width=.09\linewidth]{./big_data/observed_sequence/sequence_3/021.jpg}
\includegraphics[width=.09\linewidth]{./big_data/observed_sequence/sequence_4/021.jpg}
\includegraphics[width=.09\linewidth]{./big_data/observed_sequence/sequence_5/021.jpg}
\includegraphics[width=.09\linewidth]{./big_data/observed_sequence/sequence_6/021.jpg}
\includegraphics[width=.09\linewidth]{./big_data/observed_sequence/sequence_7/021.jpg}
\includegraphics[width=.09\linewidth]{./big_data/observed_sequence/sequence_8/021.jpg}
\includegraphics[width=.09\linewidth]{./big_data/observed_sequence/sequence_9/021.jpg}
\includegraphics[width=.09\linewidth]{./big_data/observed_sequence/sequence_10/021.jpg} \\ \vspace{2pt}
\includegraphics[width=.09\linewidth]{./big_data/observed_sequence/sequence_11/021.jpg}
\includegraphics[width=.09\linewidth]{./big_data/observed_sequence/sequence_12/021.jpg}
\includegraphics[width=.09\linewidth]{./big_data/observed_sequence/sequence_13/021.jpg}
\includegraphics[width=.09\linewidth]{./big_data/observed_sequence/sequence_14/021.jpg}
\includegraphics[width=.09\linewidth]{./big_data/observed_sequence/sequence_15/021.jpg}
\includegraphics[width=.09\linewidth]{./big_data/observed_sequence/sequence_16/021.jpg}
\includegraphics[width=.09\linewidth]{./big_data/observed_sequence/sequence_17/021.jpg}
\includegraphics[width=.09\linewidth]{./big_data/observed_sequence/sequence_18/021.jpg}
\includegraphics[width=.09\linewidth]{./big_data/observed_sequence/sequence_19/021.jpg}
\includegraphics[width=.09\linewidth]{./big_data/observed_sequence/sequence_20/021.jpg} \\ \vspace{2pt}
\includegraphics[width=.09\linewidth]{./big_data/observed_sequence/sequence_21/021.jpg}
\includegraphics[width=.09\linewidth]{./big_data/observed_sequence/sequence_22/021.jpg}
\includegraphics[width=.09\linewidth]{./big_data/observed_sequence/sequence_23/021.jpg}
\includegraphics[width=.09\linewidth]{./big_data/observed_sequence/sequence_24/021.jpg}
\includegraphics[width=.09\linewidth]{./big_data/observed_sequence/sequence_25/021.jpg}
\includegraphics[width=.09\linewidth]{./big_data/observed_sequence/sequence_26/021.jpg}
\includegraphics[width=.09\linewidth]{./big_data/observed_sequence/sequence_27/021.jpg}
\includegraphics[width=.09\linewidth]{./big_data/observed_sequence/sequence_28/021.jpg}
\includegraphics[width=.09\linewidth]{./big_data/observed_sequence/sequence_29/021.jpg}
\includegraphics[width=.09\linewidth]{./big_data/observed_sequence/sequence_30/021.jpg}\\  \vspace{-3pt} {\footnotesize (a) $21$-st frame of 30 observed sequences} \\ \vspace{3pt}

%\includegraphics[width=.09\linewidth]{./big_data/iter_2400/synthesis_1/image_0021.jpg}
%\includegraphics[width=.09\linewidth]{./big_data/iter_2400/synthesis_2/image_0021.jpg}
%\includegraphics[width=.09\linewidth]{./big_data/iter_2400/synthesis_3/image_0021.jpg}
%\includegraphics[width=.09\linewidth]{./big_data/iter_2400/synthesis_4/image_0021.jpg}
%\includegraphics[width=.09\linewidth]{./big_data/iter_2400/synthesis_5/image_0021.jpg}
%\includegraphics[width=.09\linewidth]{./big_data/iter_2400/synthesis_6/image_0021.jpg}
%\includegraphics[width=.09\linewidth]{./big_data/iter_2400/synthesis_7/image_0021.jpg}
%\includegraphics[width=.09\linewidth]{./big_data/iter_2400/synthesis_8/image_0021.jpg}
%\includegraphics[width=.09\linewidth]{./big_data/iter_2400/synthesis_9/image_0021.jpg}
%\includegraphics[width=.09\linewidth]{./big_data/iter_2400/synthesis_10/image_0021.jpg}\\ \vspace{2pt}
%\includegraphics[width=.09\linewidth]{./big_data/iter_2400/synthesis_11/image_0021.jpg}
%\includegraphics[width=.09\linewidth]{./big_data/iter_2400/synthesis_12/image_0021.jpg}
%\includegraphics[width=.09\linewidth]{./big_data/iter_2400/synthesis_13/image_0021.jpg}
%\includegraphics[width=.09\linewidth]{./big_data/iter_2400/synthesis_14/image_0021.jpg}
%\includegraphics[width=.09\linewidth]{./big_data/iter_2400/synthesis_15/image_0021.jpg}
%\includegraphics[width=.09\linewidth]{./big_data/iter_2400/synthesis_16/image_0021.jpg}
%\includegraphics[width=.09\linewidth]{./big_data/iter_2400/synthesis_17/image_0021.jpg}
%\includegraphics[width=.09\linewidth]{./big_data/iter_2400/synthesis_18/image_0021.jpg}
%\includegraphics[width=.09\linewidth]{./big_data/iter_2400/synthesis_19/image_0021.jpg}
%\includegraphics[width=.09\linewidth]{./big_data/iter_2400/synthesis_20/image_0021.jpg}\\ \vspace{2pt}
%\includegraphics[width=.09\linewidth]{./big_data/iter_2400/synthesis_21/image_0021.jpg}
%\includegraphics[width=.09\linewidth]{./big_data/iter_2400/synthesis_22/image_0021.jpg}
%\includegraphics[width=.09\linewidth]{./big_data/iter_2400/synthesis_23/image_0021.jpg}
%\includegraphics[width=.09\linewidth]{./big_data/iter_2400/synthesis_24/image_0021.jpg}
%\includegraphics[width=.09\linewidth]{./big_data/iter_2400/synthesis_25/image_0021.jpg}
%\includegraphics[width=.09\linewidth]{./big_data/iter_2400/synthesis_26/image_0021.jpg}
%\includegraphics[width=.09\linewidth]{./big_data/iter_2400/synthesis_27/image_0021.jpg}
%\includegraphics[width=.09\linewidth]{./big_data/iter_2400/synthesis_28/image_0021.jpg}
%\includegraphics[width=.09\linewidth]{./big_data/iter_2400/synthesis_29/image_0021.jpg}
%\includegraphics[width=.09\linewidth]{./big_data/iter_2400/synthesis_30/image_0021.jpg}
%\includegraphics[width=.09\linewidth]{./big_data/iter_2400/synthesis_31/image_0021.jpg}
%\includegraphics[width=.09\linewidth]{./big_data/iter_2400/synthesis_32/image_0021.jpg}
%\includegraphics[width=.09\linewidth]{./big_data/iter_2400/synthesis_33/image_0021.jpg}
%\includegraphics[width=.09\linewidth]{./big_data/iter_2400/synthesis_34/image_0021.jpg}
%\includegraphics[width=.09\linewidth]{./big_data/iter_2400/synthesis_35/image_0021.jpg}
%\includegraphics[width=.09\linewidth]{./big_data/iter_2400/synthesis_36/image_0021.jpg}
%\includegraphics[width=.09\linewidth]{./big_data/iter_2400/synthesis_37/image_0021.jpg}
%\includegraphics[width=.09\linewidth]{./big_data/iter_2400/synthesis_38/image_0021.jpg}
%\includegraphics[width=.09\linewidth]{./big_data/iter_2400/synthesis_39/image_0021.jpg}\\  {\footnotesize (b) 30 synthesized sequences} \\ \vspace{2pt}



%\includegraphics[width=.09\linewidth]{./big_data/iter_1350/synthesis_1/image_0021.jpg}
\includegraphics[width=.09\linewidth]{./big_data/iter_1350/synthesis_2/image_0021.jpg}
%\includegraphics[width=.09\linewidth]{./big_data/iter_1350/synthesis_3/image_0021.jpg}
\includegraphics[width=.09\linewidth]{./big_data/iter_1350/synthesis_4/image_0021.jpg}
\includegraphics[width=.09\linewidth]{./big_data/iter_1350/synthesis_5/image_0021.jpg}
\includegraphics[width=.09\linewidth]{./big_data/iter_1350/synthesis_6/image_0021.jpg}
\includegraphics[width=.09\linewidth]{./big_data/iter_1350/synthesis_7/image_0021.jpg}
\includegraphics[width=.09\linewidth]{./big_data/iter_1350/synthesis_8/image_0021.jpg}
\includegraphics[width=.09\linewidth]{./big_data/iter_1350/synthesis_9/image_0021.jpg}
\includegraphics[width=.09\linewidth]{./big_data/iter_1350/synthesis_10/image_0021.jpg}
\includegraphics[width=.09\linewidth]{./big_data/iter_1350/synthesis_11/image_0021.jpg}
\includegraphics[width=.09\linewidth]{./big_data/iter_1350/synthesis_12/image_0021.jpg}\\ \vspace{2pt}
\includegraphics[width=.09\linewidth]{./big_data/iter_1350/synthesis_13/image_0021.jpg}
%\includegraphics[width=.09\linewidth]{./big_data/iter_1350/synthesis_14/image_0021.jpg}
%\includegraphics[width=.09\linewidth]{./big_data/iter_1350/synthesis_15/image_0021.jpg}
\includegraphics[width=.09\linewidth]{./big_data/iter_1350/synthesis_16/image_0021.jpg}
\includegraphics[width=.09\linewidth]{./big_data/iter_1350/synthesis_17/image_0021.jpg}
\includegraphics[width=.09\linewidth]{./big_data/iter_1350/synthesis_18/image_0021.jpg}
%\includegraphics[width=.09\linewidth]{./big_data/iter_1350/synthesis_19/image_0021.jpg}
\includegraphics[width=.09\linewidth]{./big_data/iter_1350/synthesis_20/image_0021.jpg}
\includegraphics[width=.09\linewidth]{./big_data/iter_1350/synthesis_21/image_0021.jpg}
\includegraphics[width=.09\linewidth]{./big_data/iter_1350/synthesis_22/image_0021.jpg}
\includegraphics[width=.09\linewidth]{./big_data/iter_1350/synthesis_23/image_0021.jpg}
%\includegraphics[width=.09\linewidth]{./big_data/iter_1350/synthesis_24/image_0021.jpg}
\includegraphics[width=.09\linewidth]{./big_data/iter_1350/synthesis_25/image_0021.jpg}
%\includegraphics[width=.09\linewidth]{./big_data/iter_1350/synthesis_26/image_0021.jpg}
\includegraphics[width=.09\linewidth]{./big_data/iter_1350/synthesis_27/image_0021.jpg}\\ \vspace{2pt}
\includegraphics[width=.09\linewidth]{./big_data/iter_1350/synthesis_28/image_0021.jpg}
\includegraphics[width=.09\linewidth]{./big_data/iter_1350/synthesis_29/image_0021.jpg}
\includegraphics[width=.09\linewidth]{./big_data/iter_1350/synthesis_30/image_0021.jpg}
\includegraphics[width=.09\linewidth]{./big_data/iter_1350/synthesis_31/image_0021.jpg}
\includegraphics[width=.09\linewidth]{./big_data/iter_1350/synthesis_32/image_0021.jpg}
\includegraphics[width=.09\linewidth]{./big_data/iter_1350/synthesis_33/image_0021.jpg}
%\includegraphics[width=.09\linewidth]{./big_data/iter_1350/synthesis_34/image_0021.jpg}
%\includegraphics[width=.09\linewidth]{./big_data/iter_1350/synthesis_35/image_0021.jpg}
\includegraphics[width=.09\linewidth]{./big_data/iter_1350/synthesis_36/image_0021.jpg}
\includegraphics[width=.09\linewidth]{./big_data/iter_1350/synthesis_37/image_0021.jpg}
\includegraphics[width=.09\linewidth]{./big_data/iter_1350/synthesis_38/image_0021.jpg}
\includegraphics[width=.09\linewidth]{./big_data/iter_1350/synthesis_39/image_0021.jpg}\\  \vspace{-3pt} {\footnotesize (b) $21$-st frame of 30 synthesized sequences} \\ \vspace{3pt}


%\includegraphics[width=.09\linewidth]{./big_data/iter_1602_15chain/synthesis_1/image_0021.jpg}
%%\includegraphics[width=.09\linewidth]{./big_data/iter_1602_15chain/synthesis_2/image_0021.jpg}
%\includegraphics[width=.09\linewidth]{./big_data/iter_1602_15chain/synthesis_3/image_0021.jpg}
%\includegraphics[width=.09\linewidth]{./big_data/iter_1602_15chain/synthesis_4/image_0021.jpg}
%\includegraphics[width=.09\linewidth]{./big_data/iter_1602_15chain/synthesis_5/image_0021.jpg}
%\includegraphics[width=.09\linewidth]{./big_data/iter_1602_15chain/synthesis_6/image_0021.jpg}
%%\includegraphics[width=.09\linewidth]{./big_data/iter_1602_15chain/synthesis_7/image_0021.jpg}
%\includegraphics[width=.09\linewidth]{./big_data/iter_1602_15chain/synthesis_8/image_0021.jpg}
%\includegraphics[width=.09\linewidth]{./big_data/iter_1602_15chain/synthesis_9/image_0021.jpg}
%%\includegraphics[width=.09\linewidth]{./big_data/iter_1602_15chain/synthesis_10/image_0021.jpg}
%\includegraphics[width=.09\linewidth]{./big_data/iter_1602_15chain/synthesis_11/image_0021.jpg}
%\includegraphics[width=.09\linewidth]{./big_data/iter_1602_15chain/synthesis_12/image_0021.jpg}\\ \vspace{2pt}
%\includegraphics[width=.09\linewidth]{./big_data/iter_1602_15chain/synthesis_13/image_0021.jpg}
%\includegraphics[width=.09\linewidth]{./big_data/iter_1602_15chain/synthesis_14/image_0021.jpg}
%\includegraphics[width=.09\linewidth]{./big_data/iter_1602_15chain/synthesis_15/image_0021.jpg}
%\includegraphics[width=.09\linewidth]{./big_data/iter_1602_15chain/synthesis_16/image_0021.jpg}
%%\includegraphics[width=.09\linewidth]{./big_data/iter_1602_15chain/synthesis_17/image_0021.jpg}
%%\includegraphics[width=.09\linewidth]{./big_data/iter_1602_15chain/synthesis_18/image_0021.jpg}
%%\includegraphics[width=.09\linewidth]{./big_data/iter_1602_15chain/synthesis_19/image_0021.jpg}
%\includegraphics[width=.09\linewidth]{./big_data/iter_1602_15chain/synthesis_20/image_0021.jpg}
%\includegraphics[width=.09\linewidth]{./big_data/iter_1602_15chain/synthesis_21/image_0021.jpg}
%\includegraphics[width=.09\linewidth]{./big_data/iter_1602_15chain/synthesis_22/image_0021.jpg}
%%\includegraphics[width=.09\linewidth]{./big_data/iter_1602_15chain/synthesis_23/image_0021.jpg}
%%\includegraphics[width=.09\linewidth]{./big_data/iter_1602_15chain/synthesis_24/image_0021.jpg}
%\includegraphics[width=.09\linewidth]{./big_data/iter_1602_15chain/synthesis_25/image_0021.jpg}
%\includegraphics[width=.09\linewidth]{./big_data/iter_1602_15chain/synthesis_26/image_0021.jpg}
%\includegraphics[width=.09\linewidth]{./big_data/iter_1602_15chain/synthesis_27/image_0021.jpg}\\ \vspace{2pt}
%\includegraphics[width=.09\linewidth]{./big_data/iter_1602_15chain/synthesis_28/image_0021.jpg}
%\includegraphics[width=.09\linewidth]{./big_data/iter_1602_15chain/synthesis_29/image_0021.jpg}
%\includegraphics[width=.09\linewidth]{./big_data/iter_1602_15chain/synthesis_30/image_0021.jpg}
%\includegraphics[width=.09\linewidth]{./big_data/iter_1602_15chain/synthesis_31/image_0021.jpg}
%\includegraphics[width=.09\linewidth]{./big_data/iter_1602_15chain/synthesis_32/image_0021.jpg}
%\includegraphics[width=.09\linewidth]{./big_data/iter_1602_15chain/synthesis_33/image_0021.jpg}
%\includegraphics[width=.09\linewidth]{./big_data/iter_1602_15chain/synthesis_34/image_0021.jpg}
%\includegraphics[width=.09\linewidth]{./big_data/iter_1602_15chain/synthesis_35/image_0021.jpg}
%\includegraphics[width=.09\linewidth]{./big_data/iter_1602_15chain/synthesis_36/image_0021.jpg}
%\includegraphics[width=.09\linewidth]{./big_data/iter_1602_15chain/synthesis_37/image_0021.jpg}
%\includegraphics[width=.09\linewidth]{./big_data/iter_1602_15chain/synthesis_38/image_0021.jpg}
%\includegraphics[width=.09\linewidth]{./big_data/iter_1602_15chain/synthesis_39/image_0021.jpg}
%\includegraphics[width=.09\linewidth]{./big_data/iter_1602_15chain/synthesis_40/image_0021.jpg}
%\includegraphics[width=.09\linewidth]{./big_data/iter_1602_15chain/synthesis_41/image_0021.jpg}
%\includegraphics[width=.09\linewidth]{./big_data/iter_1602_15chain/synthesis_42/image_0021.jpg}
%\includegraphics[width=.09\linewidth]{./big_data/iter_1602_15chain/synthesis_43/image_0021.jpg}
%\includegraphics[width=.09\linewidth]{./big_data/iter_1602_15chain/synthesis_44/image_0021.jpg}
%\includegraphics[width=.09\linewidth]{./big_data/iter_1602_15chain/synthesis_45/image_0021.jpg}\\  {\footnotesize (b) 30 synthesized sequences} \\ \vspace{2pt}



\includegraphics[width=.09\linewidth]{./big_data/iter_1350/synthesis_32/image_0004.jpg}
\includegraphics[width=.09\linewidth]{./big_data/iter_1350/synthesis_32/image_0007.jpg}
\includegraphics[width=.09\linewidth]{./big_data/iter_1350/synthesis_32/image_0010.jpg}
\includegraphics[width=.09\linewidth]{./big_data/iter_1350/synthesis_32/image_0013.jpg}
\includegraphics[width=.09\linewidth]{./big_data/iter_1350/synthesis_32/image_0016.jpg}
\includegraphics[width=.09\linewidth]{./big_data/iter_1350/synthesis_32/image_0019.jpg}
\includegraphics[width=.09\linewidth]{./big_data/iter_1350/synthesis_32/image_0021.jpg}
\includegraphics[width=.09\linewidth]{./big_data/iter_1350/synthesis_32/image_0024.jpg}
\includegraphics[width=.09\linewidth]{./big_data/iter_1350/synthesis_32/image_0027.jpg}
\includegraphics[width=.09\linewidth]{./big_data/iter_1350/synthesis_32/image_0030.jpg}\\ \vspace{3pt}


\includegraphics[width=.09\linewidth]{./big_data/iter_1350/synthesis_38/image_0004.jpg}
\includegraphics[width=.09\linewidth]{./big_data/iter_1350/synthesis_38/image_0007.jpg}
\includegraphics[width=.09\linewidth]{./big_data/iter_1350/synthesis_38/image_0010.jpg}
\includegraphics[width=.09\linewidth]{./big_data/iter_1350/synthesis_38/image_0013.jpg}
\includegraphics[width=.09\linewidth]{./big_data/iter_1350/synthesis_38/image_0016.jpg}
\includegraphics[width=.09\linewidth]{./big_data/iter_1350/synthesis_38/image_0019.jpg}
\includegraphics[width=.09\linewidth]{./big_data/iter_1350/synthesis_38/image_0021.jpg}
\includegraphics[width=.09\linewidth]{./big_data/iter_1350/synthesis_38/image_0024.jpg}
\includegraphics[width=.09\linewidth]{./big_data/iter_1350/synthesis_38/image_0027.jpg}
\includegraphics[width=.09\linewidth]{./big_data/iter_1350/synthesis_38/image_0030.jpg}\\  \vspace{-3pt} {\footnotesize (c) 2 examples of synthesized sequences} \\ \vspace{3pt}
	\caption{Learning  from 30 observed fire videos with mini-batch implementation.}
	\label{fig:flames}
\end{center}
\end{figure}



\subsection{Experiment 3: Generating action patterns without spatial or temporal stationarity}

Experiments 1 and 2 show  that the generative spatial-temporal ConvNet can learn from sequences without alignment. We can also specialize it to learning roughly aligned video sequences of action patterns, which are non-stationary in either spatial or temporal domain, by using a single
top-layer filter that covers the whole video sequence. We learn a 2-layer spatial-temporal generative  ConvNet from video sequences of aligned actions. The first layer has 200 $7 \times 7  \times 7$ filters with sub-sampling size of 3 pixels and frames. The second layer is a fully connected layer with a single filter that covers the whole sequence. The observed sequences are of the size $100 \times 200 \times 70$.

Figure \ref{fig:action} displays two results of modeling and synthesizing actions from roughly  aligned video sequences. We learn a model for each category, where the number of training sequences  is 5 for the running cow example, and 2 for the running tiger example. The videos are collected from the Internet and each has 70 frames. For each example, Figure \ref{fig:action} displays segments of 2 observed sequences, and segments of 2 synthesized action sequences generated by the learning algorithm. We run $\tilde{M}=8$ paralleled  chains for the experiment of running cows, and 4 paralleled chains for the experiment of running tigers. The experiments show that our model can capture non-stationary action patterns.  
%Synthesizing action patterns can be difficult for the method of \cite{doretto2003dynamic}.

One limitation of our model is that it does not involve explicit tracking  of the objects and their parts. 
	  
\begin{figure}
\begin{center}
\includegraphics[width=.15\linewidth]{./action_synthesis/cow/observed_sequence/sequence_1/013.jpg}
%\includegraphics[width=.15\linewidth]{./action_synthesis/cow/observed_sequence/sequence_1/014.jpg}
%\includegraphics[width=.15\linewidth]{./action_synthesis/cow/observed_sequence/sequence_1/015.jpg}
\includegraphics[width=.15\linewidth]{./action_synthesis/cow/observed_sequence/sequence_1/016.jpg}
%\includegraphics[width=.15\linewidth]{./action_synthesis/cow/observed_sequence/sequence_1/017.jpg}
%\includegraphics[width=.15\linewidth]{./action_synthesis/cow/observed_sequence/sequence_1/018.jpg}
\includegraphics[width=.15\linewidth]{./action_synthesis/cow/observed_sequence/sequence_1/019.jpg}
%\includegraphics[width=.15\linewidth]{./action_synthesis/cow/observed_sequence/sequence_1/020.jpg}
%\includegraphics[width=.15\linewidth]{./action_synthesis/cow/observed_sequence/sequence_1/021.jpg}
\includegraphics[width=.15\linewidth]{./action_synthesis/cow/observed_sequence/sequence_1/022.jpg}
%\includegraphics[width=.15\linewidth]{./action_synthesis/cow/observed_sequence/sequence_1/023.jpg}
%\includegraphics[width=.15\linewidth]{./action_synthesis/cow/observed_sequence/sequence_1/024.jpg}
\includegraphics[width=.15\linewidth]{./action_synthesis/cow/observed_sequence/sequence_1/025.jpg}
%\includegraphics[width=.15\linewidth]{./action_synthesis/cow/observed_sequence/sequence_1/026.jpg}
%\includegraphics[width=.15\linewidth]{./action_synthesis/cow/observed_sequence/sequence_1/027.jpg}
\includegraphics[width=.15\linewidth]{./action_synthesis/cow/observed_sequence/sequence_1/028.jpg} \\ \vspace{1pt}
%\includegraphics[width=.15\linewidth]{./action_synthesis/cow/observed_sequence/sequence_1/029.jpg}
%\includegraphics[width=.15\linewidth]{./action_synthesis/cow/observed_sequence/sequence_1/030.jpg} \\ \vspace{3pt}

%\includegraphics[width=.15\linewidth]{./action_synthesis/cow/observed_sequence/sequence_2/013.jpg}
%\includegraphics[width=.15\linewidth]{./action_synthesis/cow/observed_sequence/sequence_2/016.jpg}                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 
%\includegraphics[width=.15\linewidth]{./action_synthesis/cow/observed_sequence/sequence_2/019.jpg}
%\includegraphics[width=.15\linewidth]{./action_synthesis/cow/observed_sequence/sequence_2/022.jpg}
%\includegraphics[width=.15\linewidth]{./action_synthesis/cow/observed_sequence/sequence_2/025.jpg}
%\includegraphics[width=.15\linewidth]{./action_synthesis/cow/observed_sequence/sequence_2/028.jpg} \\ \vspace{1pt}

\includegraphics[width=.15\linewidth]{./action_synthesis/cow/observed_sequence/sequence_3/013.jpg}
%\includegraphics[width=.15\linewidth]{./action_synthesis/cow/observed_sequence/sequence_3/014.jpg}
%\includegraphics[width=.15\linewidth]{./action_synthesis/cow/observed_sequence/sequence_3/015.jpg}
\includegraphics[width=.15\linewidth]{./action_synthesis/cow/observed_sequence/sequence_3/016.jpg}
%\includegraphics[width=.15\linewidth]{./action_synthesis/cow/observed_sequence/sequence_3/017.jpg}
%\includegraphics[width=.15\linewidth]{./action_synthesis/cow/observed_sequence/sequence_3/018.jpg}
\includegraphics[width=.15\linewidth]{./action_synthesis/cow/observed_sequence/sequence_3/019.jpg}
%\includegraphics[width=.15\linewidth]{./action_synthesis/cow/observed_sequence/sequence_3/020.jpg}
%\includegraphics[width=.15\linewidth]{./action_synthesis/cow/observed_sequence/sequence_3/021.jpg}
\includegraphics[width=.15\linewidth]{./action_synthesis/cow/observed_sequence/sequence_3/022.jpg}
%\includegraphics[width=.15\linewidth]{./action_synthesis/cow/observed_sequence/sequence_3/023.jpg}
%\includegraphics[width=.15\linewidth]{./action_synthesis/cow/observed_sequence/sequence_3/024.jpg}
\includegraphics[width=.15\linewidth]{./action_synthesis/cow/observed_sequence/sequence_3/025.jpg}
%\includegraphics[width=.15\linewidth]{./action_synthesis/cow/observed_sequence/sequence_3/026.jpg}
%\includegraphics[width=.15\linewidth]{./action_synthesis/cow/observed_sequence/sequence_3/027.jpg}
\includegraphics[width=.15\linewidth]{./action_synthesis/cow/observed_sequence/sequence_3/028.jpg}
\\ \vspace{-3pt} {\footnotesize observed sequences} \\ \vspace{3pt}
%\includegraphics[width=.15\linewidth]{./action_synthesis/cow/observed_sequence/sequence_3/029.jpg}
%\includegraphics[width=.15\linewidth]{./action_synthesis/cow/observed_sequence/sequence_3/030.jpg} 

\includegraphics[width=.15\linewidth]{./action_synthesis/cow/synthesis_chain_1/image_0013.jpg}
%\includegraphics[width=.15\linewidth]{./action_synthesis/cow/synthesis_chain_1/image_0014.jpg}
%\includegraphics[width=.15\linewidth]{./action_synthesis/cow/synthesis_chain_1/image_0015.jpg}
\includegraphics[width=.15\linewidth]{./action_synthesis/cow/synthesis_chain_1/image_0016.jpg}
%\includegraphics[width=.15\linewidth]{./action_synthesis/cow/synthesis_chain_1/image_0017.jpg}
%\includegraphics[width=.15\linewidth]{./action_synthesis/cow/synthesis_chain_1/image_0018.jpg}
\includegraphics[width=.15\linewidth]{./action_synthesis/cow/synthesis_chain_1/image_0019.jpg}
%\includegraphics[width=.15\linewidth]{./action_synthesis/cow/synthesis_chain_1/image_0020.jpg}
%\includegraphics[width=.15\linewidth]{./action_synthesis/cow/synthesis_chain_1/image_0021.jpg}
\includegraphics[width=.15\linewidth]{./action_synthesis/cow/synthesis_chain_1/image_0022.jpg}
%\includegraphics[width=.15\linewidth]{./action_synthesis/cow/synthesis_chain_1/image_0023.jpg}
%\includegraphics[width=.15\linewidth]{./action_synthesis/cow/synthesis_chain_1/image_0024.jpg}
\includegraphics[width=.15\linewidth]{./action_synthesis/cow/synthesis_chain_1/image_0025.jpg}
%\includegraphics[width=.15\linewidth]{./action_synthesis/cow/synthesis_chain_1/image_0026.jpg}
%\includegraphics[width=.15\linewidth]{./action_synthesis/cow/synthesis_chain_1/image_0027.jpg}
\includegraphics[width=.15\linewidth]{./action_synthesis/cow/synthesis_chain_1/image_0028.jpg} \\ \vspace{1pt}


%\includegraphics[width=.15\linewidth]{./action_synthesis/cow/synthesis_chain_8/image_0013.jpg}
%\includegraphics[width=.15\linewidth]{./action_synthesis/cow/synthesis_chain_8/image_0016.jpg}
%\includegraphics[width=.15\linewidth]{./action_synthesis/cow/synthesis_chain_8/image_0019.jpg}
%\includegraphics[width=.15\linewidth]{./action_synthesis/cow/synthesis_chain_8/image_0022.jpg}
%\includegraphics[width=.15\linewidth]{./action_synthesis/cow/synthesis_chain_8/image_0025.jpg}
%\includegraphics[width=.15\linewidth]{./action_synthesis/cow/synthesis_chain_8/image_0028.jpg}\\ \vspace{1pt}

%\includegraphics[width=.15\linewidth]{./action_synthesis/cow/synthesis_chain_1/image_0029.jpg}
%\includegraphics[width=.15\linewidth]{./action_synthesis/cow/synthesis_chain_1/image_0030.jpg} \\ \vspace{3pt}
\includegraphics[width=.15\linewidth]{./action_synthesis/cow/synthesis_chain_6/image_0013.jpg}
%\includegraphics[width=.15\linewidth]{./action_synthesis/cow/synthesis_chain_6/image_0014.jpg}
%\includegraphics[width=.15\linewidth]{./action_synthesis/cow/synthesis_chain_6/image_0015.jpg}
\includegraphics[width=.15\linewidth]{./action_synthesis/cow/synthesis_chain_6/image_0016.jpg}
%\includegraphics[width=.15\linewidth]{./action_synthesis/cow/synthesis_chain_6/image_0017.jpg}
%\includegraphics[width=.15\linewidth]{./action_synthesis/cow/synthesis_chain_6/image_0018.jpg}
\includegraphics[width=.15\linewidth]{./action_synthesis/cow/synthesis_chain_6/image_0019.jpg}
%\includegraphics[width=.15\linewidth]{./action_synthesis/cow/synthesis_chain_6/image_0020.jpg}
%\includegraphics[width=.15\linewidth]{./action_synthesis/cow/synthesis_chain_6/image_0021.jpg}
\includegraphics[width=.15\linewidth]{./action_synthesis/cow/synthesis_chain_6/image_0022.jpg}
%\includegraphics[width=.15\linewidth]{./action_synthesis/cow/synthesis_chain_6/image_0023.jpg}
%\includegraphics[width=.15\linewidth]{./action_synthesis/cow/synthesis_chain_6/image_0024.jpg}
\includegraphics[width=.15\linewidth]{./action_synthesis/cow/synthesis_chain_6/image_0025.jpg}
%\includegraphics[width=.15\linewidth]{./action_synthesis/cow/synthesis_chain_6/image_0026.jpg}
%\includegraphics[width=.15\linewidth]{./action_synthesis/cow/synthesis_chain_6/image_0027.jpg}
\includegraphics[width=.15\linewidth]{./action_synthesis/cow/synthesis_chain_6/image_0028.jpg}
%\includegraphics[width=.15\linewidth]{./action_synthesis/cow/synthesis_chain_6/image_0029.jpg}
%\includegraphics[width=.15\linewidth]{./action_synthesis/cow/synthesis_chain_6/image_0030.jpg} 
\\ \vspace{-3pt} {\footnotesize synthesized sequences} \\ \vspace{2pt} (a) running cows \vspace{5pt}\\

\includegraphics[width=.15\linewidth]{./action_synthesis/tiger/observed_sequence/sequence_1/027.jpg}
%\includegraphics[width=.15\linewidth]{./action_synthesis/tiger/observed_sequence/sequence_1/028.jpg}
%\includegraphics[width=.15\linewidth]{./action_synthesis/tiger/observed_sequence/sequence_1/029.jpg}
\includegraphics[width=.15\linewidth]{./action_synthesis/tiger/observed_sequence/sequence_1/030.jpg}
%\includegraphics[width=.15\linewidth]{./action_synthesis/tiger/observed_sequence/sequence_1/031.jpg}
%\includegraphics[width=.15\linewidth]{./action_synthesis/tiger/observed_sequence/sequence_1/032.jpg}
\includegraphics[width=.15\linewidth]{./action_synthesis/tiger/observed_sequence/sequence_1/033.jpg}
%\includegraphics[width=.15\linewidth]{./action_synthesis/tiger/observed_sequence/sequence_1/034.jpg}
%\includegraphics[width=.15\linewidth]{./action_synthesis/tiger/observed_sequence/sequence_1/035.jpg}
\includegraphics[width=.15\linewidth]{./action_synthesis/tiger/observed_sequence/sequence_1/036.jpg}
%\includegraphics[width=.15\linewidth]{./action_synthesis/tiger/observed_sequence/sequence_1/037.jpg}
%\includegraphics[width=.15\linewidth]{./action_synthesis/tiger/observed_sequence/sequence_1/038.jpg}
\includegraphics[width=.15\linewidth]{./action_synthesis/tiger/observed_sequence/sequence_1/039.jpg}
%\includegraphics[width=.15\linewidth]{./action_synthesis/tiger/observed_sequence/sequence_1/040.jpg}
%\includegraphics[width=.15\linewidth]{./action_synthesis/tiger/observed_sequence/sequence_1/041.jpg}
\includegraphics[width=.15\linewidth]{./action_synthesis/tiger/observed_sequence/sequence_1/042.jpg} \\ \vspace{1pt}
%\includegraphics[width=.15\linewidth]{./action_synthesis/tiger/observed_sequence/sequence_1/043.jpg}
%\includegraphics[width=.15\linewidth]{./action_synthesis/tiger/observed_sequence/sequence_1/044.jpg} \\ \vspace{3pt}

\includegraphics[width=.15\linewidth]{./action_synthesis/tiger/observed_sequence/sequence_2/027.jpg}
%\includegraphics[width=.15\linewidth]{./action_synthesis/tiger/observed_sequence/sequence_2/028.jpg}
%\includegraphics[width=.15\linewidth]{./action_synthesis/tiger/observed_sequence/sequence_2/029.jpg}
\includegraphics[width=.15\linewidth]{./action_synthesis/tiger/observed_sequence/sequence_2/030.jpg}
%\includegraphics[width=.15\linewidth]{./action_synthesis/tiger/observed_sequence/sequence_2/031.jpg}
%\includegraphics[width=.15\linewidth]{./action_synthesis/tiger/observed_sequence/sequence_2/032.jpg}
\includegraphics[width=.15\linewidth]{./action_synthesis/tiger/observed_sequence/sequence_2/033.jpg}
%\includegraphics[width=.15\linewidth]{./action_synthesis/tiger/observed_sequence/sequence_2/034.jpg}
%\includegraphics[width=.15\linewidth]{./action_synthesis/tiger/observed_sequence/sequence_2/035.jpg}
\includegraphics[width=.15\linewidth]{./action_synthesis/tiger/observed_sequence/sequence_2/036.jpg}
%\includegraphics[width=.15\linewidth]{./action_synthesis/tiger/observed_sequence/sequence_2/037.jpg}
%\includegraphics[width=.15\linewidth]{./action_synthesis/tiger/observed_sequence/sequence_2/038.jpg}
\includegraphics[width=.15\linewidth]{./action_synthesis/tiger/observed_sequence/sequence_2/039.jpg}
%\includegraphics[width=.15\linewidth]{./action_synthesis/tiger/observed_sequence/sequence_2/040.jpg}
%\includegraphics[width=.15\linewidth]{./action_synthesis/tiger/observed_sequence/sequence_2/041.jpg}
\includegraphics[width=.15\linewidth]{./action_synthesis/tiger/observed_sequence/sequence_2/042.jpg}\\  \vspace{-3pt} {\footnotesize observed sequences} \vspace{3pt}
%\includegraphics[width=.15\linewidth]{./action_synthesis/tiger/observed_sequence/sequence_2/043.jpg}
%\includegraphics[width=.15\linewidth]{./action_synthesis/tiger/observed_sequence/sequence_2/044.jpg} \\ {\footnotesize observed sequences} \\ \vspace{3pt}

\includegraphics[width=.15\linewidth]{./action_synthesis/tiger/synthesis_chain_4/image_0027.jpg}
%\includegraphics[width=.15\linewidth]{./action_synthesis/tiger/synthesis_chain_4/image_0028.jpg}
%\includegraphics[width=.15\linewidth]{./action_synthesis/tiger/synthesis_chain_4/image_0029.jpg}
\includegraphics[width=.15\linewidth]{./action_synthesis/tiger/synthesis_chain_4/image_0030.jpg}
%\includegraphics[width=.15\linewidth]{./action_synthesis/tiger/synthesis_chain_4/image_0031.jpg}
%\includegraphics[width=.15\linewidth]{./action_synthesis/tiger/synthesis_chain_4/image_0032.jpg}
\includegraphics[width=.15\linewidth]{./action_synthesis/tiger/synthesis_chain_4/image_0033.jpg}
%\includegraphics[width=.15\linewidth]{./action_synthesis/tiger/synthesis_chain_4/image_0034.jpg}
%\includegraphics[width=.15\linewidth]{./action_synthesis/tiger/synthesis_chain_4/image_0035.jpg}
\includegraphics[width=.15\linewidth]{./action_synthesis/tiger/synthesis_chain_4/image_0036.jpg}
%\includegraphics[width=.15\linewidth]{./action_synthesis/tiger/synthesis_chain_4/image_0037.jpg}
%\includegraphics[width=.15\linewidth]{./action_synthesis/tiger/synthesis_chain_4/image_0038.jpg}
\includegraphics[width=.15\linewidth]{./action_synthesis/tiger/synthesis_chain_4/image_0039.jpg}
%\includegraphics[width=.15\linewidth]{./action_synthesis/tiger/synthesis_chain_4/image_0040.jpg}
%\includegraphics[width=.15\linewidth]{./action_synthesis/tiger/synthesis_chain_4/image_0041.jpg}
\includegraphics[width=.15\linewidth]{./action_synthesis/tiger/synthesis_chain_4/image_0042.jpg} \\ \vspace{1pt}
%\includegraphics[width=.15\linewidth]{./action_synthesis/tiger/synthesis_chain_4/image_0043.jpg}
%\includegraphics[width=.15\linewidth]{./action_synthesis/tiger/synthesis_chain_4/image_0044.jpg} \\ \vspace{3pt}

\includegraphics[width=.15\linewidth]{./action_synthesis/tiger/synthesis_chain_2/image_0027.jpg}
%\includegraphics[width=.15\linewidth]{./action_synthesis/tiger/synthesis_chain_2/image_0028.jpg}
%\includegraphics[width=.15\linewidth]{./action_synthesis/tiger/synthesis_chain_2/image_0029.jpg}
\includegraphics[width=.15\linewidth]{./action_synthesis/tiger/synthesis_chain_2/image_0030.jpg}
%\includegraphics[width=.15\linewidth]{./action_synthesis/tiger/synthesis_chain_2/image_0031.jpg}
%\includegraphics[width=.15\linewidth]{./action_synthesis/tiger/synthesis_chain_2/image_0032.jpg}
\includegraphics[width=.15\linewidth]{./action_synthesis/tiger/synthesis_chain_2/image_0033.jpg}
%\includegraphics[width=.15\linewidth]{./action_synthesis/tiger/synthesis_chain_2/image_0034.jpg}
%\includegraphics[width=.15\linewidth]{./action_synthesis/tiger/synthesis_chain_2/image_0035.jpg}
\includegraphics[width=.15\linewidth]{./action_synthesis/tiger/synthesis_chain_2/image_0036.jpg}
%\includegraphics[width=.15\linewidth]{./action_synthesis/tiger/synthesis_chain_2/image_0037.jpg}
%\includegraphics[width=.15\linewidth]{./action_synthesis/tiger/synthesis_chain_2/image_0038.jpg}
\includegraphics[width=.15\linewidth]{./action_synthesis/tiger/synthesis_chain_2/image_0039.jpg}
%\includegraphics[width=.15\linewidth]{./action_synthesis/tiger/synthesis_chain_2/image_0040.jpg}
%\includegraphics[width=.15\linewidth]{./action_synthesis/tiger/synthesis_chain_2/image_0041.jpg}
\includegraphics[width=.15\linewidth]{./action_synthesis/tiger/synthesis_chain_2/image_0042.jpg}
%\includegraphics[width=.15\linewidth]{./action_synthesis/tiger/synthesis_chain_2/image_0043.jpg}
%\includegraphics[width=.15\linewidth]{./action_synthesis/tiger/synthesis_chain_2/image_0044.jpg} 
\\  \vspace{-3pt} {\footnotesize synthesized sequences} \\  \vspace{2pt}(b) running tigers \\[3pt]
	\caption{Synthesizing action patterns. For each action video sequence, 6 continuous frames are shown. (a) running cows. Frames of 2 of 5 training sequences are displayed. The corresponding frames of 2 of 8 synthesized sequences generated by the learning algorithm are displayed. (b) running tigers. Frames of 2 observed training sequences are displayed. The corresponding frames of 2 of 4 synthesized sequences are displayed.}
	\label{fig:action}
\end{center}
\end{figure}

\subsection{Experiment 4: Learning from incomplete data}

Our model can learn from video sequences with occluded pixels. The task is inspired by the fact that most of the videos contain occluded objects. Our learning method can be adapted to this task with minimal modification. The modification involves, for each iteration, running $k$ steps of Langevin dynamics to recover the occluded regions of the observed sequences. At each iteration, we use the  completed observed sequences and the synthesized sequences to compute the gradient of the log-likelihood and update the model parameters. Our method simultaneously accomplishes the following tasks: (1) recover the occluded pixels of the training video sequences, (2) synthesize new video sequences from the learned model, (3) learn the model by updating the model parameters using the recovered sequences and the synthesized sequences. See Algorithm \ref{code:recovery} for the  description of the learning, sampling, and recovery algorithm. 

\begin{algorithm}
\caption{Learning, sampling, and recovery algorithm}
\label{code:recovery}
\begin{algorithmic}[1]

\REQUIRE ~~\\
(1) training image sequences with occluded pixels $\{\I_m, m=1,...,M\}$ \\
(2) binary masks $\{O_m, m=1,...,M \}$ indicating the locations of the occluded pixels in the training image sequences\\  
(3) number of synthesized image sequences $\tilde{M}$\\
(4) number of Langevin steps $l$ for synthesizing image sequences\\
(5) number of Langevin steps $k$ for recovering the occluded pixels\\
(6) number of learning iterations $T$

\ENSURE~~\\
(1) estimated parameters $w$\\
(2) synthesized image sequences $\{\tI_m, m = 1, ..., \tilde{M}\}$ 
(3) recovered image sequences $\{\I^{'}_m, m=1,...,M\}$
\item[]
\STATE Let $t\leftarrow 0$, initialize $w^{(0)}$.
\STATE Initialize $\tI_m $, for $m = 1, ..., \tilde{M}$. 
\STATE Initialize $\I^{'}_m$, for $m = 1, ..., M$. 
\REPEAT 
\STATE For each $m$, run $k$ steps of Langevin dynamics to recover the occluded region of $\I^{'}_m$, i.e., starting from the current $\I^{'}_m$, each step 
follows equation (\ref{eq:Langevin}), but only the occluded pixels in $\I^{'}_m$ are updated in each step. 
\STATE For each $m$, run $l$ steps of Langevin dynamics to update $\tI_m$, i.e., starting from the current $\tI_m$, each step 
follows equation (\ref{eq:Langevin}). 
\STATE Calculate  $H^{\obs} = \sum_{m=1}^{M} \frac{\partial}{\partial w} f(\I^{'}_m; w^{(t)})/M$, and
$H^{\syn} =  \sum_{m=1}^{\tM} \frac{\partial}{\partial w} f(\tI_m; w^{(t)})/\tM$.
\STATE Update $w^{(t+1)} \leftarrow w^{(t)} + \eta ( H^{\obs} - H^{\syn}) $,  with step size $\eta$. 
\STATE Let $t \leftarrow t+1$
\UNTIL $t = T$
\end{algorithmic}
\end{algorithm}




\begin{table}[h]
\caption{Recovery errors in occlusion experiments}\label{recoveryExp}
\vskip 0.02in
\begin{center}
\begin{footnotesize}
%\begin{footnotesize}
%\vskip 0.08in
(a) salt and pepper masks\\
\begin{tabular}{|c|c|c|c|}
\hline
           & ours   & MRF-$\ell_1$      & MRF-$\ell_2$  \\ \hline \hline
flag       & \textbf{3.7923} & 6.6211  & 10.9216  \\ \hline
fountain   & \textbf{5.5403} & 8.1904  & 11.3850  \\ \hline
ocean      & \textbf{3.3739} & 7.2983 & 9.6020     \\ \hline
playing    & \textbf{5.9035} & 14.3665& 15.7735    \\ \hline
sea world & \textbf{5.3720} &  10.6127  & 11.7803  \\ \hline
traffic    & \textbf{7.2029} & 14.7512 & 17.6790  \\ \hline
windmill   & \textbf{5.9484} &  8.9095   & 12.6487  \\ \hline \hline
Avg.   & \textbf{5.3048} &  10.1071    & 12.8272  \\ \hline
\end{tabular}
\vskip 0.08in
(b) single region masks \\
\begin{tabular}{|c|c|c|c|}
\hline
           & ours    & MRF-$\ell_1$      & MRF-$\ell_2$  \\ \hline \hline
flag       & \textbf{8.1636}  & 10.6586  & 12.5300\\ \hline
fountain   & \textbf{6.0323}  & 11.8299  & 12.1696\\ \hline
ocean      & \textbf{3.4842}  & 8.7498 & 9.8078\\ \hline
playing    & \textbf{6.1575}  & 15.6296& 15.7085\\ \hline
sea world & \textbf{5.8850} & 12.0297  & 12.2868\\ \hline
traffic    & \textbf{6.8306}  & 15.3660 & 16.5787\\ \hline
windmill   & \textbf{7.8858} & 11.7355   & 13.2036\\ \hline \hline
Avg.   & \textbf{ 6.3484} & 12.2856    & 13.1836\\ \hline
\end{tabular}
\vskip 0.08in
(c) 50$\%$ missing frames\\
\begin{tabular}{|c|c|c|c|}
\hline
           & ours    & MRF-$\ell_1$      & MRF-$\ell_2$  \\ \hline \hline
flag       & \textbf{5.5992} & 10.7171  & 12.6317\\ \hline
fountain   & \textbf{8.0531} & 19.4331  & 13.2251\\ \hline
ocean      & \textbf{4.0428}  & 9.0838 & 9.8913\\ \hline
playing    & \textbf{7.6103 } & 22.2827& 17.5692\\ \hline
sea world & \textbf{5.4348} & 13.5101  &12.9305\\ \hline
traffic    & \textbf{8.8245}  & 16.6965 & 17.1830\\ \hline
windmill   & \textbf{7.5346} & 13.3364  & 12.9911 \\ \hline \hline
Avg.   & \textbf{ 6.7285} & 15.0085   & 13.7746\\ \hline
\end{tabular}
%\end{footnotesize}
\end{footnotesize}
\end{center}
\end{table}






\begin{figure}[h]
\begin{center}

%%\includegraphics[width=.15\linewidth]{.//recovery/windmill/observed/001.jpg}
%\includegraphics[width=.15\linewidth]{.//recovery/windmill/observed/006.jpg}
%%\includegraphics[width=.15\linewidth]{.//recovery/windmill/observed/011.jpg}
%\includegraphics[width=.15\linewidth]{.//recovery/windmill/observed/016.jpg}
%%\includegraphics[width=.15\linewidth]{.//recovery/windmill/observed/021.jpg}
%\includegraphics[width=.15\linewidth]{.//recovery/windmill/observed/026.jpg}
%%\includegraphics[width=.15\linewidth]{.//recovery/windmill/observed/031.jpg}
%\includegraphics[width=.15\linewidth]{.//recovery/windmill/observed/036.jpg}
%%\includegraphics[width=.15\linewidth]{.//recovery/windmill/observed/041.jpg}
%\includegraphics[width=.15\linewidth]{.//recovery/windmill/observed/046.jpg}
%%\includegraphics[width=.15\linewidth]{.//recovery/windmill/observed/051.jpg} \\[3px]
%\includegraphics[width=.15\linewidth]{.//recovery/windmill/observed/056.jpg} \\ \vspace{3pt}

\includegraphics[width=.15\linewidth]{.//recovery/windmill/masked/001.jpg}
%\includegraphics[width=.15\linewidth]{.//recovery/windmill/masked/006.jpg}
\includegraphics[width=.15\linewidth]{.//recovery/windmill/masked/011.jpg}
%\includegraphics[width=.15\linewidth]{.//recovery/windmill/masked/016.jpg}
\includegraphics[width=.15\linewidth]{.//recovery/windmill/masked/021.jpg}
%\includegraphics[width=.15\linewidth]{.//recovery/windmill/masked/026.jpg}
\includegraphics[width=.15\linewidth]{.//recovery/windmill/masked/031.jpg}
%\includegraphics[width=.15\linewidth]{.//recovery/windmill/masked/036.jpg}
\includegraphics[width=.15\linewidth]{.//recovery/windmill/masked/041.jpg}
%\includegraphics[width=.15\linewidth]{.//recovery/windmill/masked/046.jpg}
\includegraphics[width=.15\linewidth]{.//recovery/windmill/masked/051.jpg} 
%\includegraphics[width=.15\linewidth]{.//recovery/windmill/masked/056.jpg} \\ \vspace{3pt}
\\[3px]
\includegraphics[width=.15\linewidth]{.//recovery/windmill/recovery/image_0001.jpg}
%\includegraphics[width=.15\linewidth]{.//recovery/windmill/recovery/6_image.png}
\includegraphics[width=.15\linewidth]{.//recovery/windmill/recovery/image_0011.jpg}
%\includegraphics[width=.15\linewidth]{.//recovery/windmill/recovery/16_image.png}
\includegraphics[width=.15\linewidth]{.//recovery/windmill/recovery/image_0021.jpg}
%\includegraphics[width=.15\linewidth]{.//recovery/windmill/recovery/26_image.png}
\includegraphics[width=.15\linewidth]{.//recovery/windmill/recovery/image_0031.jpg}
%\includegraphics[width=.15\linewidth]{.//recovery/windmill/recovery/36_image.png}
\includegraphics[width=.15\linewidth]{.//recovery/windmill/recovery/image_0041.jpg}
%\includegraphics[width=.15\linewidth]{.//recovery/windmill/recovery/46_image.png}
\includegraphics[width=.15\linewidth]{.//recovery/windmill/recovery/image_0051.jpg}
%\includegraphics[width=.15\linewidth]{.//recovery/windmill/recovery/56_image.png} 
\\ (a) $50\%$ salt and pepper masks \\[3px]

%\includegraphics[width=.15\linewidth]{.//recovery/ocean/observed/002.jpg}
%\includegraphics[width=.15\linewidth]{.//recovery/ocean/observed/012.jpg}
%\includegraphics[width=.15\linewidth]{.//recovery/ocean/observed/022.jpg}
%\includegraphics[width=.15\linewidth]{.//recovery/ocean/observed/032.jpg}
%\includegraphics[width=.15\linewidth]{.//recovery/ocean/observed/042.jpg}
%\includegraphics[width=.15\linewidth]{.//recovery/ocean/observed/052.jpg} \\ [3px]
\includegraphics[width=.15\linewidth]{.//recovery/ocean/masked/002.jpg}
\includegraphics[width=.15\linewidth]{.//recovery/ocean/masked/012.jpg}
\includegraphics[width=.15\linewidth]{.//recovery/ocean/masked/022.jpg}
\includegraphics[width=.15\linewidth]{.//recovery/ocean/masked/032.jpg}
\includegraphics[width=.15\linewidth]{.//recovery/ocean/masked/042.jpg}
\includegraphics[width=.15\linewidth]{.//recovery/ocean/masked/052.jpg} \\ [3px]
\includegraphics[width=.15\linewidth]{.//recovery/ocean/recovery/image_0002.jpg}
\includegraphics[width=.15\linewidth]{.//recovery/ocean/recovery/image_0012.jpg}
\includegraphics[width=.15\linewidth]{.//recovery/ocean/recovery/image_0022.jpg}
\includegraphics[width=.15\linewidth]{.//recovery/ocean/recovery/image_0032.jpg}
\includegraphics[width=.15\linewidth]{.//recovery/ocean/recovery/image_0042.jpg}
\includegraphics[width=.15\linewidth]{.//recovery/ocean/recovery/image_0052.jpg}
\\ (b) single region masks \\[3px]



%%\includegraphics[width=.15\linewidth]{.//recovery/traffic/observed/010.jpg}
%\includegraphics[width=.15\linewidth]{.//recovery/traffic/observed/006.jpg}
%%\includegraphics[width=.15\linewidth]{.//recovery/traffic/observed/020.jpg}
%\includegraphics[width=.15\linewidth]{.//recovery/traffic/observed/016.jpg}
%%\includegraphics[width=.15\linewidth]{.//recovery/traffic/observed/030.jpg}
%\includegraphics[width=.15\linewidth]{.//recovery/traffic/observed/026.jpg}
%%\includegraphics[width=.15\linewidth]{.//recovery/traffic/observed/040.jpg}
%\includegraphics[width=.15\linewidth]{.//recovery/traffic/observed/036.jpg}
%%\includegraphics[width=.15\linewidth]{.//recovery/traffic/observed/050.jpg}
%\includegraphics[width=.15\linewidth]{.//recovery/traffic/observed/046.jpg}
%%\includegraphics[width=.15\linewidth]{.//recovery/traffic/observed/060.jpg}
%\includegraphics[width=.15\linewidth]{.//recovery/traffic/observed/056.jpg} \\ \vspace{3pt} 
%%\\[3px]

\includegraphics[width=.15\linewidth]{.//recovery/traffic/masked/010.jpg}
%\includegraphics[width=.15\linewidth]{.//recovery/traffic/masked/006.jpg}
\includegraphics[width=.15\linewidth]{.//recovery/traffic/masked/020.jpg}
%\includegraphics[width=.15\linewidth]{.//recovery/traffic/masked/016.jpg}
\includegraphics[width=.15\linewidth]{.//recovery/traffic/masked/030.jpg}
%\includegraphics[width=.15\linewidth]{.//recovery/traffic/masked/026.jpg}
\includegraphics[width=.15\linewidth]{.//recovery/traffic/masked/040.jpg}
%\includegraphics[width=.15\linewidth]{.//recovery/traffic/masked/036.jpg}
\includegraphics[width=.15\linewidth]{.//recovery/traffic/masked/050.jpg}
%\includegraphics[width=.15\linewidth]{.//recovery/traffic/masked/046.jpg}
\includegraphics[width=.15\linewidth]{.//recovery/traffic/masked/060.jpg}
%\includegraphics[width=.15\linewidth]{.//recovery/traffic/masked/056.jpg} \\ \vspace{3pt}
\\[3px]
\includegraphics[width=.15\linewidth]{.//recovery/traffic/recovery/image_0010.jpg}
%\includegraphics[width=.15\linewidth]{.//recovery/traffic/recovery/6_image.png}
\includegraphics[width=.15\linewidth]{.//recovery/traffic/recovery/image_0020.jpg}
%\includegraphics[width=.15\linewidth]{.//recovery/traffic/recovery/16_image.png}
\includegraphics[width=.15\linewidth]{.//recovery/traffic/recovery/image_0030.jpg}
%\includegraphics[width=.15\linewidth]{.//recovery/traffic/recovery/26_image.png}
\includegraphics[width=.15\linewidth]{.//recovery/traffic/recovery/image_0040.jpg}
%\includegraphics[width=.15\linewidth]{.//recovery/traffic/recovery/36_image.png}
\includegraphics[width=.15\linewidth]{.//recovery/traffic/recovery/image_0050.jpg}
%\includegraphics[width=.15\linewidth]{.//recovery/traffic/recovery/46_image.png}
\includegraphics[width=.15\linewidth]{.//recovery/traffic/recovery/image_0060.jpg}
%\includegraphics[width=.15\linewidth]{.//recovery/traffic/recovery/56_image.png} 
\\ (c) $50\%$ missing frames \vspace{3pt}
	\caption{Learning from occluded video sequences. For each experiment, the first row shows a segment of the occluded sequence with black masks. The second row shows the corresponding segment of the recovered sequence. (a) type 1: salt and pepper mask.  (b) type 2: single region mask. (c) type 3: missing frames.}
	\label{fig:recovery}
\end{center}
\end{figure}

%\begin{table*}[]
%\centering
%\caption{Recovery errors on the single region mask experiments}
%\label{tab:region}
%\begin{tabular}{|c|c|c|c|c|c|c|c|}
%\hline
%           & ours    & 1       & 2       & 4       & 5       & MRF-L1-2D & MRF-L2-2D\\ \hline
%\hline
%flag       & 9.8137  & 24.5996 & 22.3153 & 22.2656 & 22.2900 & 31.3721& 29.1582\\ \hline
%fountain   & 7.4072  & 24.5299 & 25.8164 & 26.106  & 26.1413 &35.6409 &31.8588\\ \hline
%ocean      & 4.6218  & 26.4170 & 23.7262 & 23.7418 & 23.7756 & 28.4418& 27.8158\\ \hline
%playing    & 9.2957  & 47.4262 & 30.1308 & 30.1986 & 30.2640 & 33.0756& 33.0135\\ \hline
%sea\_world & 8.2168  & 31.3763 & 25.3196 & 25.317  & 25.3424 & 28.8703 & 29.2586\\ \hline
%traffic    & 9.5381  & 52.8235 & 37.6525 & 37.6366 & 37.7162 & 40.5750 &40.6746\\ \hline
%windmill   & 10.1710 & 29.3174 & 26.4172 & 26.4288 & 26.4599 & 31.1005 &31.6339\\ \hline \hline
%Avg.       &   ?      &    ?     &       ?  &      ?   &    ? & & \\ \hline
%\end{tabular}
%\end{table*}
  
%\begin{table}[]
%\centering
%\caption{Recovery errors on the missing frames experiments}
%\label{tab:missing}
%\begin{tabular}{|c|c|c|c|c|c|}
%\hline
%           & ours    & 4D      & 3D \\ \hline
%flag       & 6.1095  & 6.1818  & 4.1967\\ \hline
%fountain   & 9.6798  & 5.7215  & 4.9426\\ \hline
%ocean      & 5.1456  & 6.7555  & 3.1661\\ \hline
%playing    & 11.5997  & 9.3978 & 7.7901\\ \hline
%sea\_world & 8.0797 & 9.0523   & 5.5439\\ \hline
%traffic    & 11.2457  & 10.3282 & 9.7068\\ \hline
%windmill   & 8.2731 & 8.8873    & 5.6250\\ \hline
%Avg.   & ? & ?    & \\ \hline
%\end{tabular}
%\end{table}



\begin{figure}[h]
\begin{center}
%\includegraphics[width=.3\linewidth]{.//background_inpainting/original_images/065.jpg}
%\includegraphics[width=.3\linewidth]{.//background_inpainting/masked_images/065.jpg}
%\includegraphics[width=.3\linewidth]{.//background_inpainting/results/65_image.png}\\[3px]
%
%\includegraphics[width=.3\linewidth]{.//background_inpainting/original_images/072.jpg}
%\includegraphics[width=.3\linewidth]{.//background_inpainting/masked_images/072.jpg}
%\includegraphics[width=.3\linewidth]{.//background_inpainting/results/72_image.png}\\ [3px]
%
%\includegraphics[width=.3\linewidth]{.//background_inpainting/original_images/079.jpg}
%\includegraphics[width=.3\linewidth]{.//background_inpainting/masked_images/079.jpg}
%\includegraphics[width=.3\linewidth]{.//background_inpainting/results/79_image.png}\\[3px]
%
%
%
%\includegraphics[width=.3\linewidth]{.//background_inpainting/original_images/086.jpg}
%\includegraphics[width=.3\linewidth]{.//background_inpainting/masked_images/086.jpg}
%\includegraphics[width=.3\linewidth]{.//background_inpainting/results/86_image.png}\\ [3px]
%
%
%\includegraphics[width=.3\linewidth]{.//background_inpainting/original_images/093.jpg}
%\includegraphics[width=.3\linewidth]{.//background_inpainting/masked_images/093.jpg}
%\includegraphics[width=.3\linewidth]{.//background_inpainting/results/93_image.png}\\ [3px]
%
%\includegraphics[width=.3\linewidth]{.//background_inpainting/original_images/100.jpg}
%\includegraphics[width=.3\linewidth]{.//background_inpainting/masked_images/100.jpg}
%\includegraphics[width=.3\linewidth]{.//background_inpainting/results/100_image.png}
%\\

\includegraphics[width=.27\linewidth]{.//background_inpainting/boat/original/060.jpg}
\includegraphics[width=.27\linewidth]{.//background_inpainting/boat/masked/060.jpg}
\includegraphics[width=.27\linewidth]{.//background_inpainting/boat/results/image_0060.jpg}\\[3px]

%\includegraphics[width=.28\linewidth]{.//background_inpainting/boat/original/080.jpg}
%\includegraphics[width=.28\linewidth]{.//background_inpainting/boat/masked/080.jpg}
%\includegraphics[width=.28\linewidth]{.//background_inpainting/boat/results/image_0080.jpg}\\[3px]

\includegraphics[width=.27\linewidth]{.//background_inpainting/boat/original/100.jpg}
\includegraphics[width=.27\linewidth]{.//background_inpainting/boat/masked/100.jpg}
\includegraphics[width=.27\linewidth]{.//background_inpainting/boat/results/image_0100.jpg}\\
(a) removing a moving boat in the lake\\[3px]

\includegraphics[width=.27\linewidth]{.//background_inpainting/chatelet/original/040.jpg}
\includegraphics[width=.27\linewidth]{.//background_inpainting/chatelet/masked/040.jpg}
\includegraphics[width=.27\linewidth]{.//background_inpainting/chatelet/results/0040.png}\\[3px]

%\includegraphics[width=.28\linewidth]{.//background_inpainting/chatelet/original/065.jpg}
%\includegraphics[width=.28\linewidth]{.//background_inpainting/chatelet/masked/065.jpg}
%\includegraphics[width=.28\linewidth]{.//background_inpainting/chatelet/results/0065.png}\\[3px]

\includegraphics[width=.27\linewidth]{.//background_inpainting/chatelet/original/090.jpg}
\includegraphics[width=.27\linewidth]{.//background_inpainting/chatelet/masked/090.jpg}
\includegraphics[width=.27\linewidth]{.//background_inpainting/chatelet/results/0090.png}\\
(b) removing a walking person in front of fountain\\[3px]
	\caption{Background inpainting for videos. For each experiment, the first column displays 2 frames of the original video. The second column shows the corresponding frames with black masks occluding the target to be removed. The third column shows the inpainting result by our algorithm. (a) moving boat. (b) walking person.}
	\label{fig:bg_inpainting}
\end{center}
\end{figure}




%
%
%\begin{table}[tb]
%\centering
%{\small
%\caption{Recovery errors on the single region mask experiments}
%\label{tab:region}
%\begin{tabular}{|c|c|c|c|c|c|}
%\hline
%           & ours    & MRF-L1-3D      & MRF-L2-3D  \\ \hline \hline
%flag       & \textbf{9.8137}  & 10.6586  & 12.5300\\ \hline
%fountain   & \textbf{7.4072}  & 11.8299  & 12.1696\\ \hline
%ocean      & \textbf{4.6218}  & 8.7498 & 9.8078\\ \hline
%playing    & \textbf{9.2957}  & 15.6296& 15.7085\\ \hline
%sea\_world & \textbf{8.2168} &  12.0297  & 12.2868\\ \hline
%traffic    & \textbf{9.5381}  & 15.3660 & 16.5787\\ \hline
%windmill   & \textbf{10.1710} &  11.7355   & 13.2036\\ \hline \hline
%Avg.   & \textbf{8.4378} & 12.2856    & 13.1836\\ \hline
%\end{tabular}
%}
%\end{table}
%
%
%\begin{table}[!tb]
%
%\centering
%{\small
%\caption{Recovery errors on the missing frames experiments}
%\label{tab:missing}
%\begin{tabular}{|c|c|c|c|c|c|}
%\hline
%           & ours    & MRF-L1-3D      & MRF-L2-3D  \\ \hline \hline
%flag       & \textbf{6.1095}  & 10.7171  & 12.6317\\ \hline
%fountain   & \textbf{9.6798}  & 19.4331  & 13.2251\\ \hline
%ocean      & \textbf{5.1456}  & 9.0838 & 9.8913\\ \hline
%playing    & \textbf{11.5997 } & 22.2827& 17.5692\\ \hline
%sea\_world & \textbf{8.0797} &  13.5101  &12.9305\\ \hline
%traffic    & \textbf{11.2457}  & 16.6965 & 17.183\\ \hline
%windmill   & \textbf{8.2731} &   13.3364  & 12.9911\\ \hline \hline
%Avg.   & \textbf{8.5904} & 12.2856   & 13.7746\\ \hline
%\end{tabular}
%}
%\end{table}


%\begin{figure*}[tb]
%\begin{center}
%\includegraphics[width=.13\linewidth]{.//background_inpainting/original_images/065.jpg}
%\includegraphics[width=.13\linewidth]{.//background_inpainting/original_images/072.jpg}
%\includegraphics[width=.13\linewidth]{.//background_inpainting/original_images/079.jpg}
%\includegraphics[width=.13\linewidth]{.//background_inpainting/original_images/086.jpg}
%\includegraphics[width=.13\linewidth]{.//background_inpainting/original_images/093.jpg}
%\includegraphics[width=.13\linewidth]{.//background_inpainting/original_images/100.jpg}
%%\includegraphics[width=.15\linewidth]{.//background_inpainting/original_images/071.jpg}
%%\includegraphics[width=.15\linewidth]{.//background_inpainting/original_images/072.jpg}
%%\includegraphics[width=.15\linewidth]{.//background_inpainting/original_images/073.jpg}
%%\includegraphics[width=.15\linewidth]{.//background_inpainting/original_images/074.jpg}
%%\includegraphics[width=.15\linewidth]{.//background_inpainting/original_images/075.jpg}
%%\includegraphics[width=.15\linewidth]{.//background_inpainting/original_images/076.jpg}
%
%\includegraphics[width=.13\linewidth]{.//background_inpainting/masked_images/065.jpg}
%\includegraphics[width=.13\linewidth]{.//background_inpainting/masked_images/072.jpg}
%\includegraphics[width=.13\linewidth]{.//background_inpainting/masked_images/079.jpg}
%\includegraphics[width=.13\linewidth]{.//background_inpainting/masked_images/086.jpg}
%\includegraphics[width=.13\linewidth]{.//background_inpainting/masked_images/093.jpg}
%\includegraphics[width=.13\linewidth]{.//background_inpainting/masked_images/100.jpg}
%%\includegraphics[width=.15\linewidth]{.//background_inpainting/masked_images/071.jpg}
%%\includegraphics[width=.16\linewidth]{.//background_inpainting/masked_images/072.jpg}
%%\includegraphics[width=.16\linewidth]{.//background_inpainting/masked_images/073.jpg}
%%\includegraphics[width=.16\linewidth]{.//background_inpainting/masked_images/074.jpg}
%%\includegraphics[width=.16\linewidth]{.//background_inpainting/masked_images/075.jpg}
%%\includegraphics[width=.16\linewidth]{.//background_inpainting/masked_images/076.jpg}
%
%\includegraphics[width=.13\linewidth]{.//background_inpainting/results/65_image.png}
%\includegraphics[width=.13\linewidth]{.//background_inpainting/results/72_image.png}
%\includegraphics[width=.13\linewidth]{.//background_inpainting/results/79_image.png}
%\includegraphics[width=.13\linewidth]{.//background_inpainting/results/86_image.png}
%\includegraphics[width=.13\linewidth]{.//background_inpainting/results/93_image.png}
%\includegraphics[width=.13\linewidth]{.//background_inpainting/results/100_image.png}
%%\includegraphics[width=.15\linewidth]{.//background_inpainting/results/71_image.png}
%%\includegraphics[width=.16\linewidth]{.//background_inpainting/results/72_image.png}
%%\includegraphics[width=.16\linewidth]{.//background_inpainting/results/73_image.png}
%%\includegraphics[width=.16\linewidth]{.//background_inpainting/results/74_image.png}
%%\includegraphics[width=.16\linewidth]{.//background_inpainting/results/75_image.png}
%%\includegraphics[width=.16\linewidth]{.//background_inpainting/results/76_image.png}
%\\
%	\caption{Background inpainting for videos. The first row displays 6 image frames of the original video. The second row shows the corresponding image frames with black masks occluding the target to be removed. The third row shows the inpainting result by our algorithm.}
%	\label{fig:bg_inpainting}
%\end{center}
%\end{figure*}

We design 3 types of occlusions: (1) Type 1: salt and pepper occlusion, where we randomly place $7 \times 7$ masks on the $150 \times 150$ image domain to cover $50\%$ of the pixels of the videos. (2) Type 2: single region mask occlusion, where we randomly place a $60 \times 60$ mask on the $150 \times 150$ image domain. (3) Type 3: missing frames, where we randomly block $50\%$ of the image frames from each video. Figure \ref{fig:recovery} displays one example of the recovery result for each type of occlusion. Each video has 70 frames. 

To quantitatively evaluate the qualities of the recovered videos, we test our method on 7 video sequences, which are collected from DynTex++ dataset of \cite{ghanem2010maximum}, with 3 types of occlusions. We use the same model structure as the one used in Experiment 3. The number of Langevin steps for recovering is set to be equal to the number of Langevin steps for synthesizing, which is 20. For each experiment, we report the recovery errors measured by the average per pixel difference between the original image sequence and the recovered image sequence on the occluded pixels. The range of pixel intensities is $[0,255]$. We compare our results with the results obtained by a generic Markov random field model defined on the video sequence. The model is a 3D (spatial-temporal) Markov random field, whose potentials are pairwise $\ell_1$ or $\ell_2$ differences between nearest neighbor pixels, where the nearest neighbors are defined in both the spatial and temporal domains. The image sequences are recovered by sampling the intensities of the occluded pixels conditional on the observed pixels using the Gibbs sampler. Table \ref{recoveryExp} shows the comparison results for 3 types of occlusions. We can see that our model can recover the incomplete data, while learning from them.   


\subsection{Experiment 5: Background inpainting}


If a moving object in the video is occluded in each frame, it turns out that the recovery algorithm will become an algorithm for background inpainting of videos, where the goal is to remove the undesired moving object from the video. We use the same model as the one in Experiment 2  for Figure \ref{fig:DTresults}. Figure \ref{fig:bg_inpainting} shows two examples of removals of (a) a moving boat and (b) a walking person  respectively. The videos are collected from \cite{data}. For each example, the first column displays 2 frames of the original video. The second column shows the corresponding frames with masks occluding the target to be removed. The third column presents the inpainting result by our algorithm. The video size is $130 \times 174 \times 150$ in example (a) and $130 \times 230 \times 104$ in example (b). The experiment is different from the video inpainting by interpolation. We synthesize image patches to fill in the empty regions of the video by running Langevin dynamics. For both Experiments 4 and 5, we run a single Langevin chain for synthesis.       



\section{Conclusion}

In this paper, we propose a spatial-temporal generative ConvNet model for synthesizing dynamic patterns, such as dynamic textures and action patterns. Our experiments show that the model can synthesize realistic dynamic patterns. 
Moreover, it is possible to learn the model from video sequences with occluded pixels or missing frames.  

Other experiments, not included in this paper, show that our method can also generate sound patterns. 

The MCMC sampling of  the model can be sped up by learning and sampling the models at multiple scales, or by recruiting the generator network to reconstruct and regenerate the synthesized examples as in cooperative training \cite{xie2016cooperative}. 

\section*{Acknowledgments}
{\noindent{The work is supported by NSF DMS 1310391, DARPA SIMPLEX N66001-15-C-4035,  ONR MURI N00014-16-1-2007, and  DARPA ARO W911NF-16-1-0579. }}

%\newpage

{\small
\bibliographystyle{ieee}
\bibliography{mybibfile}
}

\end{document}
