Joint Inference in Weakly-Annotated Image Datasets via Dense Correspondence

Abstract

We present a principled framework for inferring pixel labels in weakly-annotated image datasets. Most previous, example-based approaches to computer vision rely on a large corpus of densely labeled images. However, for large, modern image datasets, such labels are expensive to obtain and are often unavailable. We establish a large-scale graphical model spanning all labeled and unlabeled images, then solve it to infer pixel labels jointly for all images in the dataset while enforcing consistent annotations over similar visual patterns. This model requires significantly less labeled data and assists in resolving ambiguities by propagating inferred annotations from images with stronger local visual evidences to images with weaker local evidences. We apply our proposed framework to two computer vision problems, namely image annotation with semantic segmentation, and object discovery and co-segmentation (segmenting multiple images containing a common object). Extensive numerical evaluations and comparisons show that our method consistently outperforms the state-of-the-art in automatic annotation and semantic labeling, while requiring significantly less labeled data. In contrast to previous co-segmentation techniques, our method manages to discover and segment objects well even in the presence of substantial amounts of noise images (images not containing the common object), as typical for datasets collected from Internet search.

Path-Invariant Map Networks

“…In this paper, we focus on a map synchronization setting, where matrix-based map encodings become too costly or even infeasible. Such instances include optimizing dense flows across many high-resolution images [30,24,41] or optimizing a network of neural networks, each of which maps one domain to another domain (e.g., 3D semantic segmentation [12] maps the space of 3D scenes to the space of 3D segmentations). In this setting, maps are usually encoded as broadly defined parametric maps (e.g., feedforward neural networks), and map optimization reduces to optimizing hyper-parameters and/or network parameters.…”

Joint Supervoxel Classification Forest for Weakly-Supervised Organ Segmentation

“…Related Work: In the computer vision community, there has been considerable research done in the domain of weakly supervised image segmentation [10,6,2], segmentation propagation [13,7], and co-segmentation [17], where the minimal assumption is that a common object is present in all of the images. Other form of weak annotations could be included such as bounding boxes, scribbles or tags indicating the presence of some object of interest.…”

“…One issue when attempting to apply some of the methods to medical images is scalability, where the main bottleneck is obtaining correspondences between the images. Some state-of-the-art methods [13] rely on dense pixel-wise correspondences, which is infeasible to apply to a large dataset of 3D medical images. In an attempt to overcome such issues, other methods advocate using superpixels in an image as a building block in unsupervised and weakly supervised segmentation [14,18,7], where feature descriptors are typically computed on a pixel-level and then aggregated within superpixels; however, descriptor choice is non-trivial and can still be computationally costly for 3D images depending on the type of features.…”