Agreement, the F-Measure, and Reliability in Information Retrieval

Abstract

Information retrieval studies that involve searching the Internet or marking phrases usually lack a well-defined number of negative cases. This prevents the use of traditional interrater reliability metrics like the kappa statistic to assess the quality of expert-generated gold standards. Such studies often quantify system performance as precision, recall, and F-measure, or as agreement. It can be shown that the average F-measure among pairs of experts is numerically identical to the average positive specific agreement among experts and that kappa approaches these measures as the number of negative cases grows large. Positive specific agreement-or the equivalent F-measure-may be an appropriate way to quantify interrater reliability and therefore to assess the reliability of a gold standard in these studies.

A Methodology towards Effective and Efficient Manual Document Annotation: Addressing Annotator Discrepancy and Annotation Quality

“…However, it is not suitable for entity recognition tasks [26]. We adopt the F-measure proposed by [13], which allows computing pair-wise inter-annotator agreement using the standard Precision, Recall and the harmonic F-measure in information studies by treating one annotator as gold standard and the other as predictions. Table 1 shows the pair-wise agreement for each entity class.…”

Detecting drugs and adverse events from Spanish social media streams

“…This metric approximates the kappa coefficient (Cohen, 1960) 8 http://labda.inf.uc3m.es/SpanishADRCorpus when the number of true negatives (TN) is very large (Hripcsak and Rothschild, 2005). In our case, we can state that the number of TN is very high since TN are all the terms that are not true positives, false positives nor false negatives.…”

Inforex—a Collaborative System for Text Corpora Annotation and Analysis

“…The current version of Inforex enables simultaneous and independent annotation of the same text sample by more than one annotator. Moreover, the annotation process coordinator may keep track of inter-annotator agreement between two raters thanks to the Agreement module which uses Positive Specific Agreement (PSA) measure (Hripcsak and Rothschild, 2005) to calculate the reliability (see Figure 5). View configuration gives the opportunity to define annotation layers, subsets or categories, users and set of documents that have to be analysed.…”