A comparison of statistical significance tests for information retrieval evaluation

Smucker, Mark D.; Allan, James; Carterette, Ben

doi:10.1145/1321440.1321528

Cited by 553 publications

(318 citation statements)

References 21 publications

Supporting

Mentioning

308

Contrasting

Unclassified

Order By: Relevance

“…Differences of acylcarnitine markers between groups were tested for significance by randomised one-way ANOVA (Edgington 1995;Howell 2001;Smucker et al 2007). For analysis of acylcarnitines in relation to time of sampling, curve fitting and nonlinear regression were performed using R (Spiess 2012;Elzhov et al 2013).…”

Section: Discussionmentioning

confidence: 99%

Medium-Chain Acyl-CoA Dehydrogenase Deficiency: Evaluation of Genotype-Phenotype Correlation in Patients Detected by Newborn Screening

et al. 2015

View full text Add to dashboard Cite

Background: Medium-chain acyl-CoA dehydrogenase deficiency (MCADD) is included in many newborn screening programmes worldwide. In addition to the prevalent mutation c.985A>G in the ACADM gene, potentially mild mutations like c.199T>C are frequently found in screening cohorts. There is ongoing discussion whether this mutation is associated with a clinical phenotype.Methods: In 37 MCADD patients detected by newborn screening, biochemical phenotype (octanoylcarnitine (C8), ratios of C8 to acetylcarnitine (C2), decanoylcarnitine (C10) and dodecanoylcarnitine (C12) at screening and confirmation) and clinical phenotype (inpatient emergency treatment, metabolic decompensations, clinical assessments, psychometric tests) were assessed in relation to genotype.

show abstract

Section: Discussionmentioning

confidence: 99%

Medium-Chain Acyl-CoA Dehydrogenase Deficiency: Evaluation of Genotype-Phenotype Correlation in Patients Detected by Newborn Screening

et al. 2015

View full text Add to dashboard Cite

show abstract

“…We chose a one-tailed non-parametric randomisation test (i.e. permutation test) due to its robustness in information retrieval as shown by Smucker et al (2007). 6 We performed the test using 100,000 random samples with a confidence interval value of 95% (i.e.…”

Section: Average R-precisionmentioning

confidence: 99%

A comparison of code similarity analysers

2017

View full text Add to dashboard Cite

Copying and pasting of source code is a common activity in software engineering. Often, the code is not copied as it is and it may be modified for various purposes; e.g. refactoring, bug fixing, or even software plagiarism. These code modifications could affect the performance of code similarity analysers including code clone and plagiarism detectors to some certain degree. We are interested in two types of code modification in this study: pervasive modifications, i.e. transformations that may have a global effect, and local modifications, i.e. code changes that are contained in a single method or code block. We evaluate 30 code similarity detection techniques and tools using five experimental scenarios for Java source code. These are (1) pervasively modified code, created with tools for source code and bytecode obfuscation, and boiler-plate code, (2) source code normalisation through compilation and decompilation using different decompilers, (3) reuse of optimal configurations over different data sets, (4) tool evaluation using ranked-based measures, and (5) local + global code modifications. Our experimental results show that in the presence of pervasive modifications, some of the general textual similarity measures can offer similar performance to specialised code similarity tools, whilst in the presence of boiler-plate code, highly specialised source code similarity detection techniques and tools outperform textual similarity measures. Our study strongly validates the use of compilation/decompilation as a normalisation technique. Its use reduced false classifications to zero for three of the tools. Moreover, we demonstrate that optimal configurations are very sensitive to a specific data set. After directly applying optimal configurations derived from one data set to another, the tools perform poorly on the new data set. The code similarity analysers are thoroughly evaluated not only based on several well-known pair-based and query-based error measures but also on each specific type of pervasive code modification. This broad, thorough study is the largest in existence and potentially an invaluable guide for future users of similarity detection in source code.

show abstract

“…Does consolidation improve federated search? We assess the ranking using Normalized Discounted Cumulative Gain (NDCG) and report the results in Table 6a for the movie and in Table 6b for the publication scenario and indicate statistically significant improvements using Fisher's two-sided, paired randomization test [19]. Systems.…”

Section: Ranking Evaluationmentioning

confidence: 99%

Federated Entity Search Using On-the-Fly Consolidation

Herzig

Mika

Blanco

et al. 2013

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Abstract. Nowadays, search on the Web goes beyond the retrieval of textual Web sites and increasingly takes advantage of the growing amount of structured data. Of particular interest is entity search, where the units of retrieval are structured entities instead of textual documents. These entities reside in different sources, which may provide only limited information about their content and are therefore called "uncooperative". Further, these sources capture complementary but also redundant information about entities. In this environment of uncooperative data sources, we study the problem of federated entity search, where redundant information about entities is reduced on-the-fly through entity consolidation performed at query time. We propose a novel method for entity consolidation that is based on using language models and completely unsupervised, hence more suitable for this on-the-fly uncooperative setting than state-of-the-art methods that require training data. Further, we apply the same language model technique to deal with the federated search problem of ranking results returned from different sources. Particular novel are the mechanisms we propose to incorporate consolidation results into this ranking. We perform experiments using real Web queries and data sources. Our experiments show that our approach for federated entity search with on-the-fly consolidation improves upon the performance of a state-of-the-art preference aggregation baseline and also benefits from consolidation.

show abstract

A comparison of statistical significance tests for information retrieval evaluation

Cited by 553 publications

References 21 publications

Medium-Chain Acyl-CoA Dehydrogenase Deficiency: Evaluation of Genotype-Phenotype Correlation in Patients Detected by Newborn Screening

Medium-Chain Acyl-CoA Dehydrogenase Deficiency: Evaluation of Genotype-Phenotype Correlation in Patients Detected by Newborn Screening

A comparison of code similarity analysers

Federated Entity Search Using On-the-Fly Consolidation

Contact Info

Product

Resources

About