Show, Match and Segment: Joint Weakly Supervised Learning of Semantic Matching and Object Co-Segmentation

Abstract

We present an approach for jointly matching and segmenting object instances of the same category within a collection of images. In contrast to existing algorithms that tackle the tasks of semantic matching and object co-segmentation in isolation, our method exploits the complementary nature of the two tasks. The key insights of our method are two-fold. First, the estimated dense correspondence field from semantic matching provides supervision for object co-segmentation by enforcing consistency between the predicted masks from a pair of images. Second, the predicted object masks from object co-segmentation in turn allow us to reduce the adverse effects due to background clutters for improving semantic matching. Our model is end-to-end trainable and does not require supervision from manually annotated correspondences and object masks. We validate the efficacy of our approach on four benchmark datasets: TSS, Internet, PF-PASCAL, and PF-WILLOW, and show that our algorithm performs favorably against the state-of-the-art methods on both semantic matching and object co-segmentation tasks.Index Terms-Semantic matching, object co-segmentation, weakly-supervised learning. !Our work. In this paper, we propose to jointly tackle both semantic matching and object co-segmentation with a twostream network in an end-to-end trainable fashion. Our key insights are two-fold. First, to suppress the effect of background clutters, the predicted object masks by object co-segmentation allow the model to focus on matching the segmented foreground regions while excluding background matching. Second, the estimated dense correspondence fields by semantic matching provide supervision for enforcing the model to generate geometrically consistent object masks across images. Therefore, we exploit the interdependency between the two network outputs, i.e., the estimated dense correspondence fields and the predicted foreground object masks, by introducing the cross-network consistency loss. Incorporating this loss improves both tasks since it encourages arXiv:1906.05857v1 [cs.CV]

A Review of Recent Advances in Image Co-Segmentation Techniques

“…To further understand the image co-segmentation frameworks, we compared the co-segmentation performance of nine techniques: geometric mean saliency based cosegmentation (GMS) [121], group saliency propagation based co-segmentation (GSP) [122], saliency co-fusion based co-segmentation (SCF) [76], co-segmentation and cosketch (CSST) [89], co-segmentation and co-skeletonization (CSSL) [90], joint object discovery and segmentation (JODS) [21], decomposition multiple foreground co-segmentation (DMFC) [18], joint semantic matching and co-segmentation (JSMC) [100] and multiple random walkers based cosegmentation (MRW) [47] on iCoseg dataset, MSRC dataset, Internet dataset and Coseg-Rep dataset, respectively. The experimental results are produced by directly running the implementation codes from their websites.…”

PGA-SiamNet: Pyramid Feature-Based Attention-Guided Siamese Network for Remote Sensing Orthoimagery Building Change Detection

“…The neighborhood consensus module [61] is used to obtain correlations of the two given features f a and f b , because it has already been applied and achieved superior performance in previous research [59]. COSNet [62] proposed another method which uses an affinity matrix to denote co-attention, and so to mine the correlations through adding a weight matrix and verifying the proposed three matrix styles by experiments. In the paper, the co-attention style is exploited to obtain the correlation map like COSNet, which is showed in Figure 4 with the blue dashed box.…”

Semi-Global Context Network for Semantic Correspondence

“…Inspired by RANSAC, Rocco et al [14] further introduced an end-to-end trainable and weakly supervised CNN architecture using soft inlier counts. Their work was further improved by introducing joint learning with co-segmentation [15] and applying methods for predicting the foreground region and enforcing cycle consistency [18]. Kim et al [19] designed a network with a recurrent structure to estimate geometric transformation iteratively.…”