Cooperative Training of Descriptor and Generator Networks

Figure: (a) The flow chart for training the descriptor net. The updating in Step D2 is based on the difference between the observed examples and the synthetic examples. The Langevin sampling of the synthetic examples from the current model in Step D1 can be time consuming. (b) The flow chart for training the generator net. The updating in Step G2 is based on the observed examples and their inferred latent factors. The Langevin sampling of the latent factors from the current posterior distribution in Step G1 can be time consuming. (c) The flow chart of the cooperative learning of descriptor and generator nets. The part of the flow chart for training the descriptor is similar to Algorithm D in (a), except that the D1 Langevin sampling is initialized from the initial synthetic examples supplied by the generator. The part of the flow chart for training the generator can also be mapped to Algorithm G in (b), except that the revised synthetic examples play the role of the observed examples, and the known generated latent factors can be used as inferred latent factors (or be used to initialize the G1 Langevin sampling of the latent factors).

 

Figure 1. Generating texture patterns. For each category, the first image displays the training image, and the rest are 3 of the images generated by the CoopNets algorithm.

Figure 2. Generating object patterns. Each row displays one object experiment, where the first 3 images are 3 of the training images, and the rest are 6 of the synthesized images generated.

  

Figure 3. Left: Average softmax class probability on single Imagenet category versus the number of training images. Middle: Top 5 classification error. Right: Average pairwise structural similarity.

Figure 4. Generating scene patterns. Both observed and synthesized scene images are shown for each category. The image size is 64 × 64 pixels. The categories are from MIT places205 dataset. (a) volcano. (b) desert. (c) rock. (d) apartment building.

  

(a) observed images                                                                    (b) synthesized images

Figure 5. Images generated by CoopNets learned from 10 Imagenet scene categories. The training set consists of 1100 images randomly sampled from each category. The total number of training images is 11000.

Figure 6. Interpolation between latent vectors of the images on the two ends.

We conduct an experiment on learning CoopNets from MNIST dataset of handwritten digits. The images are grey-scale with size of 28 × 28 pixels. Figure 6 displays some synthesized examples by the learned models after training. Table 2 shows a comparison of Gaussian Parzen window log-likelihood estimates of the MNIST testing set.

  

Figure 7. Generating handwritten digits by CoopNets after training.

     

(a) observed human face images                                                        (b) observed bedroom images

     

(a) synthesized human face images                                                        (b) synthesized bedroom images

Figure 8. (a) Generating human face images (128 × 128 pixels). The synthesized images are generated by the CoopNets algorithm that learns from celebA dataset with 200K training images. (b) Generating bedroom images (256 × 256 pixels). The synthesized images are generated by the CoopNets algorithm that learns from LSUN dataset with 3,033K training images.

We conduct an experiment on learning from fully observed training images, and then testing the learned model on completing the occluded testing images. The purpose of this experiment is to check whether the learned generator model can generalize to the testing data. Figure 7 shows the qualitative results. We report the recovery errors and compare our method with 8 different image inpainting methods in Table 3.

(a) face

(b) forest road

(c) hotel room

Figure 9. Pattern completion. First row: original images. Second row: occluded images. Third row: recovered images by CoopNets. (a) face. (b) forest road. (c) hotel room.

We learn to generate video sequences by cooperative training of a spatial-temporal descriptor and a spatial-temporal generator. The spatial-temporal descriptor consists of multiple layers of spatial-temporal filters that capture spatial-temporal features at various scales of the video sequences, while the spatial-temporal generator maps the latent variables to the video sequences by multiple layers of spatial-temporal kernels. Figure 8 displays the results. To evaluate the quality of the synthesized examples, we compare our model with some baseline models for dynamic textures in terms of PSNR and structural similarity measures (SSIM) on 6 dynamic texture videos. Table 4 shows the average performances of the models over the 6 videos.

   

(a) burning fire heating a pot                                                         (b) waterfall

   

(c) flashing lights                                                                  (d) water vapour

Figure 10. Generating dynamic textures by learning spatial-temporal CoopNets. In each example, the first video is the observed example, while the rest two are the synthesized examples generated by the learned model. (a) burning fire heating a pot. (b) waterfall. (c) flashing lights. (d) water vapour.