Robust segmentation of biomedical figures for image-based document retrieval

Lopez, Luis D.; Yu, Jingyi; Tudor, Catalina O.; Arighi, Cecilia N.; Huang, Hongzhan; Vijay‐Shanker, K.; Wu, Cathy H.

doi:10.1109/bibm.2012.6392706

Cited by 7 publications

(5 citation statements)

References 19 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The caption and the figure contents are combined for partitioning the figure. Figure 2 shows a gallery of panel-subcaption pairs recovered by our approach [18].…”

Section: Document Processingmentioning

confidence: 99%

See 1 more Smart Citation

An Image-Text Approach for Extracting Experimental Evidence of Protein-Protein Interactions in the Biomedical Literature

Lopez

Arighi

et al. 2013

Proceedings of the International Conference on Bioinformatics, Computational Biology and Biomedical Informatics

Self Cite

View full text Add to dashboard Cite

Proteins are complex biological polymers that mediate virtually all cellular functions. Typically these functions are modulated by protein-protein interactions (PPI). Tremendous efforts have been made by life scientists to detect PPIs through different experimental approaches and document the results through publications. On the informatics front, however, there lacks an effective means for retrieving PPI information from published literatures. In this work we present a novel framework for identifying experimental methods employed for analyzing PPI from biomedical articles. Different from state-of-the-art approaches based only on text, we explore using the combination of attributes from figures, figure captions, and text within figures for identifying PPI experimental methods. Our work is motivated by the observation that biomedical figures often constitute direct evidence of experimental results and therefore provide complementary information to texts. We start with automatically extracting unimodal panels (subfigures) and their associated subcaptions and then classifying the subfigure into different types using a proposed hierarchical image taxonomy. Next, we combine the subfigure types with text-based features to form a hybrid feature descriptor and use it for PPI method classification. We further construct a dataset starting from a set of 2, 256 documents provided by the molecular interaction database MINT. Here we show that our new approach outperforms the text-only solution for associating figures with PPI methods.

show abstract

“…The caption and the figure contents are combined for partitioning the figure. Figure 2 shows a gallery of panel-subcaption pairs recovered by our approach [18].…”

Section: Document Processingmentioning

confidence: 99%

“…Therefore, for the current task, it is crucial to partition the figures into unimodal panels. In [18], we have developed caption-figure approach for conducting this task. In brief, we first identify the number of panels by parsing the captions and then separate the caption into corresponding subcaptions.…”

Section: Document Processingmentioning

confidence: 99%

An Image-Text Approach for Extracting Experimental Evidence of Protein-Protein Interactions in the Biomedical Literature

Lopez

Arighi

et al. 2013

Proceedings of the International Conference on Bioinformatics, Computational Biology and Biomedical Informatics

Self Cite

View full text Add to dashboard Cite

show abstract

“…This manuscript is an expansion of the article [32] published as part of IEEE International Conference on Bioinformatics and Biomedicine (BIBM) 2012.…”

Section: Declarationsmentioning

confidence: 99%

“…The publication costs of this article were funded by the NSF (National Science Foundation) 2013, University of Delaware, awarded to Jingyi Yu. This manuscript is an expansion of the article [ 32 ] published as part of IEEE International Conference on Bioinformatics and Biomedicine (BIBM) 2012.…”

Section: Declarationsmentioning

confidence: 99%

A framework for biomedical figure segmentation towards image-based document retrieval

Lopez

Arighi

et al. 2013

BMC Systems Biology

Self Cite

View full text Add to dashboard Cite

The figures included in many of the biomedical publications play an important role in understanding the biological experiments and facts described within. Recent studies have shown that it is possible to integrate the information that is extracted from figures in classical document classification and retrieval tasks in order to improve their accuracy. One important observation about the figures included in biomedical publications is that they are often composed of multiple subfigures or panels, each describing different methodologies or results. The use of these multimodal figures is a common practice in bioscience, as experimental results are graphically validated via multiple methodologies or procedures. Thus, for a better use of multimodal figures in document classification or retrieval tasks, as well as for providing the evidence source for derived assertions, it is important to automatically segment multimodal figures into subfigures and panels. This is a challenging task, however, as different panels can contain similar objects (i.e., barcharts and linecharts) with multiple layouts. Also, certain types of biomedical figures are text-heavy (e.g., DNA sequences and protein sequences images) and they differ from traditional images. As a result, classical image segmentation techniques based on low-level image features, such as edges or color, are not directly applicable to robustly partition multimodal figures into single modal panels.In this paper, we describe a robust solution for automatically identifying and segmenting unimodal panels from a multimodal figure. Our framework starts by robustly harvesting figure-caption pairs from biomedical articles. We base our approach on the observation that the document layout can be used to identify encoded figures and figure boundaries within PDF files. Taking into consideration the document layout allows us to correctly extract figures from the PDF document and associate their corresponding caption. We combine pixel-level representations of the extracted images with information gathered from their corresponding captions to estimate the number of panels in the figure. Thus, our approach simultaneously identifies the number of panels and the layout of figures.In order to evaluate the approach described here, we applied our system on documents containing protein-protein interactions (PPIs) and compared the results against a gold standard that was annotated by biologists. Experimental results showed that our automatic figure segmentation approach surpasses pure caption-based and image-based approaches, achieving a 96.64% accuracy. To allow for efficient retrieval of information, as well as to provide the basis for integration into document classification and retrieval systems among other, we further developed a web-based interface that lets users easily retrieve panels containing the terms specified in the user queries.

show abstract

“…The paper [5] describes a solution to the similar problem but located in biomedical literature. Articles in this area are often composed of multiple subfigures and may illustrate diverse methodologies or results.…”

Section: Related Workmentioning

confidence: 99%

Caption-guided patent image segmentation

Sas

Markowska–Kaczmar

Moumtzidou

2016

Annals of Computer Science and Information Systems

View full text Add to dashboard Cite

Abstract-The paper presents a method of splitting patent drawings into subimages. For the image based patent retrieval and automatic document understanding it is required to use the individual subimages that are referenced in the text of a patent document. Our method utilizes the fact that subimages have their individual captions inscribed into the compound image. To find the approximate positions of subimages, first the specific captions are localized. Then subimages are found using the empirical rules concerning the relative positions of connected components to the subimage captions. These rules are based on the common sense observation that distances between connected components belonging to the same subimage are smaller than distances between connected components belonging to various subimages and that captions are located close to the corresponding subimages. Alternatively, the image segmentation can be defined as a specific optimization problem, that is aimed on maximizing the gaps between hypothetical subimages while preserving their relations to corresponding captions. The proposed segmentation method can be treated as the approximate solution of this problem.

show abstract

Robust segmentation of biomedical figures for image-based document retrieval

Cited by 7 publications

References 19 publications

An Image-Text Approach for Extracting Experimental Evidence of Protein-Protein Interactions in the Biomedical Literature

An Image-Text Approach for Extracting Experimental Evidence of Protein-Protein Interactions in the Biomedical Literature

A framework for biomedical figure segmentation towards image-based document retrieval

Caption-guided patent image segmentation

Contact Info

Product

Resources

About