A large reproducible benchmark of ontology-based methods and word embeddings for word similarity

Lastra-Díaz, Juan J.; Goikoetxea, Josu; Taieb, Mohamed Ali Hadj; Garcı́a-Serrano, Ana; Aouicha, Mohamed Ben; Agirre, Eneko; Sánchez, David

doi:10.1016/j.is.2020.101636

Cited by 14 publications

(11 citation statements)

References 45 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Most cognitive functions, such as categorization, memory, decision-making, and reasoning, are based on human similarity and relatedness judgments between concepts. As a result, there is a large collection of human-labeled measure datasets to evaluate the degree of human-likeness from the standpoint of concept similarity and concept relatedness, particularly in the domains of natural language processing (Lastra-Diaz et al, 2021 ). To assess how well each type of representation reflects human judgments, we compute Spearman correlations between model-based similarity and human assessments, as is customary.…”

Section: The Gap Analysismentioning

confidence: 99%

Statistical Analysis of Multisensory and Text-Derived Representations on Concept Learning

Wang

Zeng

2022

Front. Comput. Neurosci.

View full text Add to dashboard Cite

When learning concepts, cognitive psychology research has revealed that there are two types of concept representations in the human brain: language-derived codes and sensory-derived codes. For the objective of human-like artificial intelligence, we expect to provide multisensory and text-derived representations for concepts in AI systems. Psychologists and computer scientists have published lots of datasets for the two kinds of representations, but as far as we know, no systematic work exits to analyze them together. We do a statistical study on them in this work. We want to know if multisensory vectors and text-derived vectors reflect conceptual understanding and if they are complementary in terms of cognition. Four experiments are presented in this work, all focused on multisensory representations labeled by psychologists and text-derived representations generated by computer scientists for concept learning, and the results demonstrate that (1) for the same concept, both forms of representations can properly reflect the concept, but (2) the representational similarity analysis findings reveal that the two types of representations are significantly different, (3) as the concreteness of the concept grows larger, the multisensory representation of the concept becomes closer to human beings than the text-derived representation, and (4) we verified that combining the two improves the concept representation.

show abstract

Section: The Gap Analysismentioning

confidence: 99%

Statistical Analysis of Multisensory and Text-Derived Representations on Concept Learning

Wang

Zeng

2022

Front. Comput. Neurosci.

View full text Add to dashboard Cite

show abstract

“…All our experiments have been recorded into a Docker virtualization image that is provided as supplementary material together with our software [40] and a detailed reproducibility protocol [41] and dataset [42] to allow the easy replication of all our methods, experiments, and results. This work is based on our previous experience developing reproducible research in a series of publications in the area, such as the experimental surveys on word similarity introduced in [43][44][45][46], whose reproducibility protocols and datasets [47,48] are detailed and independently confirmed in two companion reproducible papers [38,49], and a reproducible benchmark on semantic measures libraries for the biomedical domain [39]. Finally, we refer the reader to our previous work [37] for a very detailed review of the literature on sentence similarity measures, which is omitted herein because of the lack of room and to avoid being redundant.…”

Section: Introductionmentioning

confidence: 99%

A reproducible experimental survey on biomedical sentence similarity: a string-based method sets the state of the art

Lara-Clares¹,

Lastra-Díaz²,

Garcı́a-Serrano³

2022

Preprint

Self Cite

View full text Add to dashboard Cite

This registered report introduces the largest, and for the first time, reproducible experimental survey on biomedical sentence similarity with the following aims: (1) to elucidate the state of the art of the problem; (2) to solve some reproducibility problems preventing the evaluation of most of current methods; (3) to evaluate several unexplored sentence similarity methods; (4) to evaluate for the first time an unexplored benchmark, called Corpus-Transcriptional-Regulation (CTR); (5) to carry out a study on the impact of the pre-processing stages and Named Entity Recognition (NER) tools on the performance of the sentence similarity methods; and finally, (6) to bridge the lack of software and data reproducibility resources for methods and experiments in this line of research. Our reproducible experimental survey is based on a single software platform, which is provided with a detailed reproducibility protocol and dataset as supplementary material to allow the exact replication of all our experiments and results. In addition, we introduce a new aggregated string-based sentence similarity method, called LiBlock, together with eight variants of current ontology-based methods, and a new pre-trained word embedding model trained on the full-text articles in the PMC-BioC corpus. Our experiments show that our novel string-based measure sets the new state of the art on the sentence similarity task in the biomedical domain and significantly outperforms all the methods evaluated herein, with the only exception of one ontology-based method. Likewise, our experiments confirm that the pre-processing stages, and the choice of the NER tool for ontology-based methods, have a very significant impact on the performance of the sentence similarity methods. We also detail some drawbacks and limitations of current methods, and warn on the need of refining the current benchmarks. Finally, a noticeable finding is that our new string-based method significantly outperforms all state-of-the-art Machine Learning (ML) models evaluated herein.

show abstract

“…Our experiments will be based on our implementation and evaluation of all methods analyzed herein into a common and new software platform based on an extension of the Half-Edge Semantic Measures Library (HESML, http://hesml.lsi.uned.es ) [ 40 ], called HESML for Semantic Textual Similarity (HESML-STS), as well as their subsequent recording with the Reprozip long-term reproducibility tool [ 41 ]. This work is based on our previous experience developing reproducible research in a series of publications in the area, such as the experimental surveys on word similarity introduced in [ 42 – 45 ], whose reproducibility protocols and datasets [ 46 , 47 ] are detailed and independently confirmed in two reproducible papers [ 40 , 48 ]. The experiments in this new software platform will evaluate most of the sentence similarity methods for the biomedical domain reported in the literature, as well as a set of unexplored methods which are based on adaptations from the general language domain.…”

Section: Introductionmentioning

confidence: 99%

Protocol for a reproducible experimental survey on biomedical sentence similarity

2021

Self Cite

View full text Add to dashboard Cite

Measuring semantic similarity between sentences is a significant task in the fields of Natural Language Processing (NLP), Information Retrieval (IR), and biomedical text mining. For this reason, the proposal of sentence similarity methods for the biomedical domain has attracted a lot of attention in recent years. However, most sentence similarity methods and experimental results reported in the biomedical domain cannot be reproduced for multiple reasons as follows: the copying of previous results without confirmation, the lack of source code and data to replicate both methods and experiments, and the lack of a detailed definition of the experimental setup, among others. As a consequence of this reproducibility gap, the state of the problem can be neither elucidated nor new lines of research be soundly set. On the other hand, there are other significant gaps in the literature on biomedical sentence similarity as follows: (1) the evaluation of several unexplored sentence similarity methods which deserve to be studied; (2) the evaluation of an unexplored benchmark on biomedical sentence similarity, called Corpus-Transcriptional-Regulation (CTR); (3) a study on the impact of the pre-processing stage and Named Entity Recognition (NER) tools on the performance of the sentence similarity methods; and finally, (4) the lack of software and data resources for the reproducibility of methods and experiments in this line of research. Identified these open problems, this registered report introduces a detailed experimental setup, together with a categorization of the literature, to develop the largest, updated, and for the first time, reproducible experimental survey on biomedical sentence similarity. Our aforementioned experimental survey will be based on our own software replication and the evaluation of all methods being studied on the same software platform, which will be specially developed for this work, and it will become the first publicly available software library for biomedical sentence similarity. Finally, we will provide a very detailed reproducibility protocol and dataset as supplementary material to allow the exact replication of all our experiments and results.

show abstract

A large reproducible benchmark of ontology-based methods and word embeddings for word similarity

Cited by 14 publications

References 45 publications

Statistical Analysis of Multisensory and Text-Derived Representations on Concept Learning

Statistical Analysis of Multisensory and Text-Derived Representations on Concept Learning

A reproducible experimental survey on biomedical sentence similarity: a string-based method sets the state of the art

Protocol for a reproducible experimental survey on biomedical sentence similarity

Contact Info

Product

Resources

About