Cosegmentation and Cosketch by Unsupervised Learning

Jifeng Dai1,2, Ying Nian Wu2, Jie Zhou1 and Song-Chun Zhu2

1 Department of Automation, Tsinghua University, China 2 Department of Statistics, University of California, Los Angeles (UCLA), USA


Cosegmentation refers to the problem of segmenting multiple images simultaneously by exploiting the similarities between the foreground and background regions in these images. The key issue in cosegmentation is to align common objects between these images. To address this issue, we propose an unsupervised learning framework for cosegmentation, by coupling cosegmentation with what we call ``cosketch''. The goal of cosketch is to automatically discover a codebook of deformable shape templates shared by the input images. These shape templates capture distinct image patterns and each template is matched to similar image patches in different images. Thus the cosketch of the images helps to align foreground objects, thereby providing crucial information for cosegmentation. We present a statistical model whose energy function couples cosketch and cosegmentation. We then present an unsupervised learning algorithm that performs cosketch and cosegmentation by energy minimization. Experiments show that our method outperforms state of the art methods for cosegmentation on the challenging MSRC and iCoseg datasets. We also illustrate our method on a new dataset called Coseg-Rep where cosegmentation can be performed within a single image with repetitive patterns.


The paper can be downloaded from here


The Coseg-Rep dataset can be downloaded from here. It has 23 object categories with 572 images. Among them, 22 categories are different species of animals and flowers, and each category has 9 to 49 images. Besides, here is a special category called ``repetitive", which contains 116 natural images where similar shape patterns repeat themselves within the same image, such as tree leaves and grapes etc.


A demo video can be downloaded from here.

Some examples on MSRC Bike, Cow, Chair, Face, Flower, House, Sheep, Sign, and Tree


Code can be downloaded from here.

Contact us

For comments or questions about the algorithm please email daijifeng001 -at-