Painterly Animation with Video Content Extraction

Liang Lin, Kun Zeng, Han Lv, Yizhou Wang, Yingqing Xu, Song-Chun Zhu

Int'l Symposium on Non-Photorealistic Animation and Rendering (NPAR) Annecy, France, 2010. [PDF]

Note: this work is done at Lotus Hill Research Institute , mirror page at Liang Lin's website

We present an interactive video stylization system for transforming an input video into a painterly animation.
Although similar oil-painting effects can be generated manually by the paint-on-glass technique, such animation production is not only very laborious, but also requires considerable artistic skills. For example, the Oscar-winning animation Man and Sea took artists years to accomplish. In comparison, our interactive system allows amateur players to produce painterly animations from real-life video clips with far less time and efforts. Our better results on painterly animation benefit from: a) advanced techniques in computer vision for video parsing, and b) advanced painterly rendering techniques based on image parsing.

The system consists of two phases: A Content Extraction phase to obtain semantic objects in the video and establish dense feature correspondence; and a Painterly Rendering phase to select, place and propagate brush strokes for stylized animations based on the semantic content and object motion derived from the first phase. Our system has the following contributions: (1) It renders artistic style animation using a diverse set of example-based brush strokes, and these strokes are automatically selected according to the object classes. (2) It sticks the strokes tightly to the object surface in the animations by warping and shifting strokes in accordance with the transformation of the object with dense feature correspondence in both textured and textureless areas. (3) It reduces the scintillation effects by several techniques: (i) confining the strokes inside each object; (ii) the deferred rendering and backward completion for newly birth strokes, and (iii) a damped system to stabilize strokes in space and time.

[Representative Results]

Figure 1. A few sample frames of animations from our method. The high-resolution images can be downloaded [here].

[Stylized Animations Demos] DEMO1 [avi 19.5M] (from the Cartoon Movie: Princess Mononoke)
DEMO2 [avi 12.2M] (from the Movie: the Lord of the Rings)
DEMO3 [avi 17.6M] (from the Movie: the Lord of the Rings)
DEMO4 [avi 34.9M] (The lady sequence by a hand-held camera)
DEMO5 [avi 29.2M] (The LENA sequence originally proposed by Wang et al. 2004)