Biomedical article retrieval using multimodal features and image annotations in region-based CBIR

Abstract

Biomedical images are invaluable in establishing diagnosis, acquiring technical skills, and implementing best practices in many areas of medicine. At present, images needed for instructional purposes or in support of clinical decisions appear in specialized databases and in biomedical articles, and are often not easily accessible to retrieval tools. Our goal is to automatically annotate images extracted from scientific publications with respect to their usefulness for clinical decision support and instructional purposes, and project the annotations onto images stored in databases by linking images through content-based image similarity.Authors often use text labels and pointers overlaid on figures and illustrations in the articles to highlight regions of interest (ROI). These annotations are then referenced in the caption text or figure citations in the article text. In previous research we have developed two methods (a heuristic and dynamic time warping-based methods) for localizing and recognizing such pointers on biomedical images. In this work, we add robustness to our previous efforts by using a machine learning based approach to localizing and recognizing the pointers. Identifying these can assist in extracting relevant image content at regions within the image that are likely to be highly relevant to the discussion in the article text. Image regions can then be annotated using biomedical concepts from extracted snippets of text pertaining to images in scientific biomedical articles that are identified using National Library of Medicine's Unified Medical Language System ® (UMLS) Metathesaurus. The resulting regional annotation and extracted image content are then used as indices for biomedical article retrieval using the multimodal features and region-based content-based image retrieval (CBIR) techniques. The hypothesis that such an approach would improve biomedical document retrieval is validated through experiments on an expert-marked biomedical article dataset.

Using relevant regions in image search and query refinement for medical CBIR

“…From this segmentation, we encode a total of 69,384 regions. Next, we utilize the arrow detection algorithm from You et al, 12 which automatically detects 278 arrows from these images.…”

“…This filter is necessary because simply matching against all the possible segmented regions in our database produces too many non-relevant region matches. To accomplish this task, we build upon the work performed by You et al 12 In this work, You et al 12 noticed that images often contained pointers overlaid on figures and illustrations to highlight regions of interest. These annotations were also referenced in the captions or figure citations in the text.…”

Scalable Arrow Detection in Biomedical Images

“…From the binary image, arrow-like object separation employs a fixed sized mask (after removing the small objects and noise as in [6]), which are then used for feature computation such as major and minor axis lengths, axis ratio, area, solidity and Euler number. A recent study uses a pointer region and boundary detection to handle distorted arrows [10], which is followed by edge detection techniques and fixed thresholds as reported in [7], [9]. These candidates are used to compute overlapping regions, which are then binarized to extract the boundary of the expected pointers.…”

Automatic identification of ROI in figure images toward improving hybrid (text and image) biomedical document retrieval

“…This article presents our efforts to improve our prior work [3][4] on pointer recognition and ROI extraction to achieve better relevance quality in the proposed multimodal biomedical article retrieval. Additional pointer segmentation and ROI extraction methods were developed based on region growing method.…”

“…To improve the relevance quality of conventional retrieval approaches, we have proposed an approach using hybrid (text and image) features [3]. Information retrieval (IR) techniques are used to identify key textual features in the title, abstract, figure caption, and figure citation (mention) in the article.…”