Protocol for a reproducible experimental survey on biomedical sentence similarity

Abstract

Measuring semantic similarity between sentences is a significant task in the fields of Natural Language Processing (NLP), Information Retrieval (IR), and biomedical text mining. For this reason, the proposal of sentence similarity methods for the biomedical domain has attracted a lot of attention in recent years. However, most sentence similarity methods and experimental results reported in the biomedical domain cannot be reproduced for multiple reasons as follows: the copying of previous results without confirmation, the lack of source code and data to replicate both methods and experiments, and the lack of a detailed definition of the experimental setup, among others. As a consequence of this reproducibility gap, the state of the problem can be neither elucidated nor new lines of research be soundly set. On the other hand, there are other significant gaps in the literature on biomedical sentence similarity as follows: (1) the evaluation of several unexplored sentence similarity methods which deserve to be studied; (2) the evaluation of an unexplored benchmark on biomedical sentence similarity, called Corpus-Transcriptional-Regulation (CTR); (3) a study on the impact of the pre-processing stage and Named Entity Recognition (NER) tools on the performance of the sentence similarity methods; and finally, (4) the lack of software and data resources for the reproducibility of methods and experiments in this line of research. Identified these open problems, this registered report introduces a detailed experimental setup, together with a categorization of the literature, to develop the largest, updated, and for the first time, reproducible experimental survey on biomedical sentence similarity. Our aforementioned experimental survey will be based on our own software replication and the evaluation of all methods being studied on the same software platform, which will be specially developed for this work, and it will become the first publicly available software library for biomedical sentence similarity. Finally, we will provide a very detailed reproducibility protocol and dataset as supplementary material to allow the exact replication of all our experiments and results.

A reproducible experimental survey on biomedical sentence similarity: a string-based method sets the state of the art

“…Our main motivation is the lack of a comprehensive and reproducible experimental survey on biomedical sentence similarity that allows setting the state of the problem in a sound and reproducible way, as detailed in our previous registered report protocol [37]. Our main research questions are as follows:…”

“…The main aim of this work is to introduce a comprehensive and very detailed reproducible experimental survey of methods on biomedical sentence similarity to elucidate the state of the problem by implementing our previous registered report protocol [37]. Our experiments are based on our software implementation and evaluation of all methods analyzed herein into a common and new software platform based on an extension of the Half-Edge Semantic Measures Library (HESML) [38,39], called HESML 1 for Semantic Textual Similarity (HESML-STS).…”

“…This work is based on our previous experience developing reproducible research in a series of publications in the area, such as the experimental surveys on word similarity introduced in [43][44][45][46], whose reproducibility protocols and datasets [47,48] are detailed and independently confirmed in two companion reproducible papers [38,49], and a reproducible benchmark on semantic measures libraries for the biomedical domain [39]. Finally, we refer the reader to our previous work [37] for a very detailed review of the literature on sentence similarity measures, which is omitted herein because of the lack of room and to avoid being redundant.…”

An Integrated Platform Supporting Semantic Similarity Score Calculation and Reproducibility

“…Retrieving, reproducing and reusing SS scores for any ontology in any application is still challenging 39,44,54 . This mainly due to the lack of a tool that exhaustively implements existing SS models and related assumptions to produce consistent scores on demand and in real-time for use in related applications and for testing hypotheses.…”

“…In the context of information retrieval, these measures enable the use of fuzzy expressions based on the degree of concept similarity, producing a satisfactory performance under uncertainty. These measures are still an active research area in various domains 39,40,41 , e.g, biomedical, wordnet and arti cial intelligence.…”