HESML: a real-time semantic measures library for the biomedical domain with a reproducible survey

Abstract

Background Ontology-based semantic similarity measures based on SNOMED-CT, MeSH, and Gene Ontology are being extensively used in many applications in biomedical text mining and genomics respectively, which has encouraged the development of semantic measures libraries based on the aforementioned ontologies. However, current state-of-the-art semantic measures libraries have some performance and scalability drawbacks derived from their ontology representations based on relational databases, or naive in-memory graph representations. Likewise, a recent reproducible survey on word similarity shows that one hybrid IC-based measure which integrates a shortest-path computation sets the state of the art in the family of ontology-based semantic measures. However, the lack of an efficient shortest-path algorithm for their real-time computation prevents both their practical use in any application and the use of any other path-based semantic similarity measure. Results To bridge the two aforementioned gaps, this work introduces for the first time an updated version of the HESML Java software library especially designed for the biomedical domain, which implements the most efficient and scalable ontology representation reported in the literature, together with a new method for the approximation of the Dijkstra’s algorithm for taxonomies, called Ancestors-based Shortest-Path Length (AncSPL), which allows the real-time computation of any path-based semantic similarity measure. Conclusions We introduce a set of reproducible benchmarks showing that HESML outperforms by several orders of magnitude the current state-of-the-art libraries in the three aforementioned biomedical ontologies, as well as the real-time performance and approximation quality of the new AncSPL shortest-path algorithm. Likewise, we show that AncSPL linearly scales regarding the dimension of the common ancestor subgraph regardless of the ontology size. Path-based measures based on the new AncSPL algorithm are up to six orders of magnitude faster than their exact implementation in large ontologies like SNOMED-CT and GO. Finally, we provide a detailed reproducibility protocol and dataset as supplementary material to allow the exact replication of all our experiments and results.

A reproducible experimental survey on biomedical sentence similarity: a string-based method sets the state of the art

“…The criteria for the selection of the sentence similarity methods evaluated herein is as follows: (a) all the methods that have been evaluated in BIOSSES and MedSTS datasets; (b) a selection of methods that have not been evaluated in the biomedical domain yet; (c) a collection of new variants or adaptations of methods previously proposed for the general or biomedical domain, which are evaluated for the first time in this work, such as the WBSM-cosJ&C [30,39,44,66], WBSM-coswJ&C [30,39,44,66], WBSM-Cai [30,39,67], UBSM-cosJ&C [30,39,44,66], UBSM-coswJ&C [30,39,44,66], and UBSM-Cai [30,39,67] work. For a more detailed description of the selection criteria of the methods, we refer the reader to our registered report protocol [37].…”

“…Tables 2 and 3 detail the configuration of the string-based measures and ontology-based measures that are evaluated herein, respectively. Both WBSM and UBSM methods are evaluated in combination with the following word and concept similarity measures: Rada et al [68], Jiang&Conrath [69], and three state-of-the-art unexplored measures, called cosJ&C [39,44], coswJ&C [39,44], and Cai et al [39,67]. The word similarity measure which reports the best results is used to evaluate the COM method [30,68].…”

“…The main aim of this work is to introduce a comprehensive and very detailed reproducible experimental survey of methods on biomedical sentence similarity to elucidate the state of the problem by implementing our previous registered report protocol [37]. Our experiments are based on our software implementation and evaluation of all methods analyzed herein into a common and new software platform based on an extension of the Half-Edge Semantic Measures Library (HESML) [38,39], called HESML 1 for Semantic Textual Similarity (HESML-STS). All our experiments have been recorded into a Docker virtualization image that is provided as supplementary material together with our software [40] and a detailed reproducibility protocol [41] and dataset [42] to allow the easy replication of all our methods, experiments, and results.…”

“…All our experiments have been recorded into a Docker virtualization image that is provided as supplementary material together with our software [40] and a detailed reproducibility protocol [41] and dataset [42] to allow the easy replication of all our methods, experiments, and results. This work is based on our previous experience developing reproducible research in a series of publications in the area, such as the experimental surveys on word similarity introduced in [43][44][45][46], whose reproducibility protocols and datasets [47,48] are detailed and independently confirmed in two companion reproducible papers [38,49], and a reproducible benchmark on semantic measures libraries for the biomedical domain [39]. Finally, we refer the reader to our previous work [37] for a very detailed review of the literature on sentence similarity measures, which is omitted herein because of the lack of room and to avoid being redundant.…”

“…Thus, this later family of methods defines a framework in which we can design new variants by exploring other word similarity measures. For this reason, we propose herein the evaluation of a set of new ontology-based sentence similarity measures based on two different unexplored notions as follows: (1) the evaluation of state-of-the-art word similarity measures from the general domain [46] not evaluated in the biomedical domain yet; and (2) the evaluation of several ontology-based word similarity measures based on a recent and very efficient shortest-path algorithm, called Ancestors-based Shortest-Path Length (AncSPL) [39], which is a fast approximation of the Dijkstra's algorithm [61] for taxonomies that is introduced with the first HESML version for the biomedical domain [39]. Thus, we propose here the evaluation based on the combination of WBSM and UBSM methods with the path-based word similarity methods as follows: WBSM-Rada (M7); WBSM-cosJ&C (M9); WBSM-coswJ&C (M10); WBSM-Cai (M11); UBSM-Rada (M12); UBSM-cosJ&C (M14); UBSM-coswJ&C (M15); and UBSM-Cai (M16).…”