A structured learning framework for content-based image indexing and visual query

Abstract

Abstract. Nonspecific images in a broad domain remain a challenge for content-based image retrieval. As a typical example, consumer photos exhibit highly varied content, diverse resolutions, and inconsistent quality. The objects are usually ill-posed, occluded, and cluttered with poor lighting, focus, and exposure. Traditional image retrieval approaches face many obstacles such as semantic description of images, robust semantic object segmentation, small sampling problem, semantic gaps between low-level features and high-level semantics, etc.To manage the high diversity of images in a broad domain, we propose a structured learning framework to systematically design domain-relevant visual semantics, known as semantic support regions, to support index and query in a contentbased image retrieval system. Semantic support regions are segmentation-free image regions that exhibit semantic meanings and that can be learned statistically to span a new indexing space. They are detected from image content, reconciled across multiple resolutions, and aggregated spatially to form local semantic histograms. The resulting compact and abstract representation can support both similarity-based query and compositional visual query efficiently. The query by spatial icons (QBSI) formulation is a unique visual query language to explicitly specify visual icons and spatial extents in a Boolean expression.For empirical evaluation, we perform the learning and indexing processes of 26 semantic support regions over 2400 heterogeneous consumer photos from a single family using Support Vector Machines. We report a 27% improvement in average precision over a very high dimension feature-based approach on 24 semantic queries based on multiple examples and pooled ground truths. Last but not least, we demonstrate the usefulness of the visual query language with 15 QBSI queries that have attained high precision values at top retrieved images on the 2400 consumer images.

A Semantic Fusion Approach Between Medical Images and Reports Using UMLS

“…This notion of using a visual and semantic vocabulary to represent and index image has been applied to consumer images in [15,14]. Here, we use UMLS concepts to represent each token in the medical domain.…”

Combining Textual and Visual Ontologies to Solve Medical Multimodal Queries

“…A new visual query language, Query by Spatial Icons (QBSI), has been developed to combine pattern matching and logical inference [9]. In this paper, we extend QBSI with spatial quantifiers and apply it to medical semanticsbased retrieval where the queries are text description and the query processing is carried on image indexes based on VisMed terms.…”

“…Then the query processing of query P for any image Z is to compute the truth value µ(P,Z) using appropriate logical operators using min/max fuzzy operations. The mathematical details can be found in [9].…”

“…2 illustrates one visual example each from these 39 VisMed terms. During image indexing, the 39 trained VisMed detectors are applied to the small regions in a medical image obtained in a sliding window manner [8,9]. After multi-scale reconciliation [8,9], an image block Z in a grid-based image index contains a vector of detection confidence values of VisMed terms, T i (Z), which is aggregated (averaged) over the detection vector on smaller regions z k ,…”

“…For the query processing in ImageCLEF 2005, a query text description is manually translated into a QBSI query with the help of a visual query interface [9] that outputs an XML format to state the VisMed terms, the spatial regions, the Boolean operators, and the spatial quantifiers. As an illustration, query 02 in the Medical Image Retrieval Task "Show me x-ray images with fractures of the femur" is translated as " xray-bone whole xray-pelvis upper xray-bone-fracture whole" where "whole" and "upper" refer to the whole image and upper part of an image respectively.…”