Extracting Structured Supervision From Captions for Weakly Supervised Semantic Segmentation

Abstract

Weakly supervised semantic segmentation (WSSS) methods have received significant attention in recent years, since they can dramatically reduce the annotation costs of fully supervised alternatives. While most previous studies focused on leveraging classification labels, we explore instead the use of image captions, which can be obtained easily from the web and contain richer visual information. Existing methods for this task assigned text snippets to relevant semantic labels by simply matching class names, and then employed a model trained to localize arbitrary text in images to generate pseudoground truth segmentation masks. Instead, we propose a dedicated caption processing module to extract structured supervision from captions, consisting of improved relevant object labels, their visual attributes, and additional background categories, all of which are useful for improving segmentation quality. This module uses syntactic structures learned from text data, and semantic relations retrieved from a knowledge database, without requiring additional annotations on the specific image domain, and consequently can be extended immediately to new object categories. We then present a novel localization network, which is trained to localize only these structured labels. This strategy simplifies model design, while focusing training signals on relevant visual information. Finally, we describe a method for leveraging all types of localization maps to obtain high-quality segmentation masks, which are used to train a supervised model. On the challenging MS-COCO dataset, our method moves the state-of-the-art forward significantly for WSSS with image-level supervision by a margin of 7.6% absolute (26.7% relative) mean Intersection-over-Union, achieving 54.5% precision and 50.9% recall.

Two-Stage Pedestrian Detection Model Using a New Classification Head for Domain Generalization

“…In the last decade, deep learning has enabled significant progress in a variety of applications including object detection [1,2], face recognition [3], iris recognition [4], genetic algorithms applied to CNNs [5,6], rock lithological classification [7], trademark image retrieval [8], and semantic segmentation [9], among others. Pedestrian detection is one of the key tasks in computer vision, for which several models have been developed in the past few years [10][11][12][13][14][15][16][17][18][19].…”

Iris Recognition Using Low-Level CNN Layers Without Training and Single Matching

“…With a deep learning approach, filters are no longer created by human experts, but, rather, an optimization process is performed to find the best coefficients, using a training process [20]- [22]. In addition, it is common to train a classifier, such as a Support Vector Machine (SVM), Multi Perceptron Layer (MPL), or Random Forest (RF), with a training stage [14], [23], [24].…”