Exploring Resources for Lexical Chaining: A Comparison of Automated Semantic Relatedness Measures and Human Judgments

Abstract

In the past decade various semantic relatedness, similarity, and distance measures have been proposed which play a crucial role in many NLP-applications. Researchers compete for better algorithms (and resources to base the algorithms on), and often only few percentage points seem to suffice in order to prove a new measure (or resource) more accurate than an older one. However, it is still unclear which of them performs best under what conditions. In this work we therefore present a study comparing various relatedness measures. We evaluate them on the basis of a human judgment experiment and also examine several practical issues, such as run time and coverage. We show that the performance of all measures -as compared to human estimates -is still mediocre and argue that the definition of a shared task might bring us considerably closer to results of high quality.

CLAD: A corpus-derived Chinese Lexical Association Database

“…Setting human judgments as the golden standard gives the assessment of how “good” or “bad” a measure is by its congruence with human performance. These sorts of assessments are currently the most commonly employed (Cramer, Wandmacher, & Waltinger, 2011; Johns & Jones, 2010; Recchia & Jones, 2009).…”

Effects of domain on measures of semantic relatedness

“…Metrics for the automated estimation of semantic relatedness have long been used across a wide range of applications for purposes ranging from improving document retrieval to measuring text coherence. Due to the wide applicability of relatedness measures, work has been carried out exclusively in an attempt to improve the quality of estimation with respect to specific tasks such as topic clustering (Newman, Lau, Grieser, & Baldwin, ), spelling error detection (Budanitsky & Hirst, ), or lexical chaining (Cramer et al., ) Some of this work has specifically investigated the various factors that can influence automated relatedness estimation. For example, Sanchez, Batet, Isern, and Valls () studied subsumption relationships as a feature and demonstrated improved estimation when that factor is taken into consideration.…”

“…For example, semantic relatedness has been used for analyzing query logs (Benz, Krause, Kumar, Hotho, & Stumme, 2009), clustering web pages (Yazdani & Popescu-Belis, 2011) and web services (Liu & Wong, 2009), and improving document retrieval (Muller & Gurevych, 2009;Wittek, Daranyi, & Tan, 2009). In language technology research, semantic-relatedness measures have been used for named entity recognition (Gentile, Zhang, Xia, & Iria, 2010), word-sense disambiguation (Budanitsky & Hirst, 2006;Pedersen, Banerjee, & Patwardhan, 2005), measuring text coherence (Lapata & Barzilay, 2005), document classification (Syed, Finin, & Joshi, 2008), lexical chaining (Cramer, Wandmacher, & Waltinger, 2012), or simply clustering words in general (Wong, Liu, & Bennamoun, 2008). The wide applicability of these measures has prompted a number of researchers to investigate ways to improve the correlation of automatic measurement with human judgement (Gabrilovich & Markovitch, 2007;He et al, 2011;Kulkarni & Caragea, 2009;Liu et al, 2012;Wang, Chen, & Liu, 2008).…”