Co-localization in Real-World Images

Abstract

In this paper, we tackle the problem of co-localization in real-world images. Co-localization is the problem of simultaneously localizing (with bounding boxes) objects of the same class across a set of distinct images. Although similar problems such as co-segmentation and weakly supervised localization have been previously studied, we focus on being able to perform co-localization in real-world settings, which are typically characterized by large amounts of intraclass variation, inter-class diversity, and annotation noise. To address these issues, we present a joint image-box formulation for solving the co-localization problem, and show how it can be relaxed to a convex quadratic program which can be efficiently solved. We perform an extensive evaluation of our method compared to previous state-of-theart approaches on the challenging PASCAL VOC 2007 and Object Discovery datasets. In addition, we also present a large-scale study of co-localization on ImageNet, involving ground-truth annotations for 3,624 classes and approximately 1 million images.

Efficient Image and Video Co-localization with Frank-Wolfe Algorithm

“…Our work builds upon the formulation introduced in [47] for co-localization in images, which defines an optimization objective that draws inspiration from works in image segmentation [42] and discriminative clustering [3,21,52,54]. Extending their work, we introduce a formulation for co-localization in videos that incorporates constraints and terms that capture temporal consistency, a key property in videos.…”

“…-Formulation for video co-localization. We present a novel formulation for video co-localization, extending [47] with temporal terms and constraints. -Frank-Wolfe algorithm for efficient optimization.…”

“…Co-localization is the problem of localizing (with bounding boxes) the common object in a set of images or videos. Recent work has studied co-localization in images with potentially noisy labels [47], and co-localization in videos [37] for learning object detectors. Building upon the success of a recent state-of-the-art method [47], we propose a formulation for co-localization in videos that can take advantage of temporal consistency with temporal terms and constraints, while still maintaining a standard quadratic programming formulation.…”

“…Recent work has studied co-localization in images with potentially noisy labels [47], and co-localization in videos [37] for learning object detectors. Building upon the success of a recent state-of-the-art method [47], we propose a formulation for co-localization in videos that can take advantage of temporal consistency with temporal terms and constraints, while still maintaining a standard quadratic programming formulation. We also show how we can combine both models to perform joint image-video co-localization, the logical way of utilizing all of the weakly supervised data we have available.…”

“…In addition to pixels, these images and videos are often tagged with the visual concepts they contain, leading to a natural source of weakly labeled data. Recent research has studied ways of leveraging this data, such as weakly supervised localization [13,17,32,[43][44][45][46], co-segmentation [21,23,38], and co-localization [37,47].…”