Synthesizing Dynamic Patterns by Spatial-Temporal Generative ConvNet

This repository contains a tensorflow implementation for the paper "Synthesizing Dynamic Patterns by Spatial-Temporal
Generative ConvNet".

1. Requirements

We use tensorflow(version >= 1.0) in Python 2.7 as the main implementation framework. We use additional Python packages
for reading and saving images. To install, run:

pip install -r requirements.txt

We also use ffmpeg for reading and generating videos. To install, run:

sudo sh ffmpeg_installer.sh

2. Training
Our current code has been tested on Ubuntu 14.04. We include two different optimizing approaches Adam and Stochastic
Gradient Descent for training.

2.1. Adam Optimizer
We include three different descriptor networks in our Adam/src/model.py

|    model   |                     description                     |
|------------| ----------------------------------------------------|
|            | Spatial-temporal model for generating dynamic       |
|     ST     | textures with both spatial and temporal stationarity| 
|            | (Exp 1, image size 100)                             |
|------------| ----------------------------------------------------|
|            | Spatial fully-connected model for generating dynamic|
|    FC_S    | textures with only temporal stationarity (Exp 2,    |
|            | image size 100)                                     |
|------------| ----------------------------------------------------|
|            | Spatial fully-connected model for generating dynamic|
| FC_S_large | textures with only temporal stationarity (Exp 2,    |
|            | image size 224, as described in paper)              |

To synthesize 3 different outputs of fire pot with single video input(top 70 frames):

python Adam/main.py -data_path ./trainingVideo/data_synthesis -category fire_pot -output_dir ./output -sz 224 -num_chain 3 \
-batch_size 1 -lr 0.001 -num_frames 70

The synthesized videos will be stored in output/fire_pot/final_results and images for each frame will be stored in
./output/fire_pot/synthesized_sequence.

2.2 SGD Optimizer

We also include training approach using SGD Optimizer with layer-wise learning rate. To train with Spatial-Temporal
configuration, run

python SGD/main.py -data_path ./trainingVideo/data_synthesis -category fire_pot -output_dir ./output -sz 100 -num_chain 3 \
-batch_size 1  -num_frames 70

3. Reference

    @inproceedings{stgconvnet,
        author = {Xie, Jianwen and Zhu, Song-Chun and Wu, Ying Nian},
        title = {Synthesizing Dynamic Patterns by Spatial-Temporal Generative ConvNet},
        booktitle = {The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
        month = {July},
        year = {2017}
    } 
