The Reliability of Peer Reviews of Papers on Information Systems

Wood, Michael; Roberts, Martyn; Howell, Barbara

doi:10.1177/0165551504041673

Cited by 22 publications

(22 citation statements)

References 17 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Values of Kappa between 0 and 0.2 indicate only slight agreement. The only paper we found in the field of information science [24] also reported low levels of reliability in two conferences: one conference had kappa = -0.04, the other had kappa = 0.30.…”

Section: Related Researchmentioning

confidence: 77%

See 1 more Smart Citation

Can we evaluate the quality of software engineering experiments?

Kitchenham

Sjøberg

Brereton

et al. 2010

Proceedings of the 2010 ACM-IEEE International Symposium on Empirical Software Engineering and Measurement

View full text Add to dashboard Cite

Context:The authors wanted to assess whether the quality of published human-centric software engineering experiments was improving. This required a reliable means of assessing the quality of such experiments. Aims: The aims of the study were to confirm the usability of a quality evaluation checklist, determine how many reviewers were needed per paper that reports an experiment, and specify an appropriate process for evaluating quality. Method: With eight reviewers and four papers describing human-centric software engineering experiments, we used a quality checklist with nine questions. We conducted the study in two parts: the first was based on individual assessments and the second on collaborative evaluations. Results: The inter-rater reliability was poor for individual assessments but much better for joint evaluations. Four reviewers working in two pairs with discussion were more reliable than eight reviewers with no discussion. The sum of the nine criteria was more reliable than individual questions or a simple overall assessment. Conclusions: If quality evaluation is critical, more than two reviewers are required and a round of discussion is necessary. We advise using quality criteria and basing the final assessment on the sum of the aggregated criteria. The restricted number of papers used and the relatively extensive expertise of the reviewers limit our results. In addition, the results of the second part of the study could have been affected by removing a time restriction on the review as well as the consultation process.

show abstract

Section: Related Researchmentioning

confidence: 77%

“…However, the majority of studies have looked at peer review of journal or conference papers (see, for example, [24], [20]) or the extent to which reviewers agree on whether to accept or reject research grant applications or research fellowships (see, for example, [16]). …”

Section: Related Researchmentioning

confidence: 99%

Can we evaluate the quality of software engineering experiments?

Kitchenham

Sjøberg

Brereton

et al. 2010

Proceedings of the 2010 ACM-IEEE International Symposium on Empirical Software Engineering and Measurement

View full text Add to dashboard Cite

show abstract

“…In developing our rating procedures, we look for guidance from the literature examining peer review of manuscripts (see, e.g., Strayhorn, McDermott, and Tanguay 1993, van Rooyen, Black, and Godlee 1999, Wood, Roberts and Howell 2004). In our study, raters are expected to provide objective assessments of the quality of the answers.…”

Section: Analysis and Resultsmentioning

confidence: 99%

Knowledge Market Design: A Field Experiment at Google Answers

Yan

Kim

2010

J Public Economic Theory

108

View full text Add to dashboard Cite

In a field experiment at Google Answers, we investigate the performance of price-based online knowledge markets by systematically manipulating prices. Specifically, we study the effects of price, tip, and a reputation system on both an answerer's effort and answer quality by posting real reference questions from the Internet Public Library on Google Answers under different pricing schemes. We find that a higher price leads to a significantly longer, but not better, answer, while an answerer with a higher reputation provides significantly better answers. Our results highlight the limitation of monetary incentives and the importance of reputation systems in knowledge market design. Copyright � 2010 Wiley Periodicals, Inc..

show abstract

“…Values of Kappa between 0 and 0.2 indicate only slight agreement. The only paper addressing peer review that we found in the field of information science [45] also reported low levels of reliability in reviewing performed for two conferences: one conference had Kappa = -0.04, the other had Kappa = 0.30.…”

Section: Related Researchmentioning

confidence: 82%

Three empirical studies on the agreement of reviewers about the quality of software engineering experiments

Kitchenham

Sjøberg

Dybå

et al. 2012

Information and Software Technology

View full text Add to dashboard Cite

Context: During systematic literature reviews it is necessary to assess the quality of empirical papers. Current guidelines suggest that two researchers should independently apply a quality checklist and any disagreements must be resolved. [Type text]However, there is little empirical evidence concerning the effectiveness of these guidelines. Aims: This paper investigates the three techniques that can be used to improve the reliability (i.e. the consensus among reviewers) of quality assessments, specifically, the number of reviewers, the use of a set of evaluation criteria and consultation among reviewers. We undertook a series of studies to investigate these factors. Method: Two studies involved four research papers and eight reviewers using a quality checklist with nine questions. The first study was based on individual assessments, the second study on joint assessments with a period of inter-rater discussion. A third more formal randomised block experiment involved 48 reviewers assessing two of the papers used previously in teams of one, two and three persons to assess the impact of discussion among teams of different size using the evaluations of the "teams" of one person as a control. Results: For the first two studies, the inter-rater reliability was poor for individual assessments, but better for joint evaluations. However, the results of the third study contradicted the results of study 2. Inter-rater reliability was poor for all groups but worse for teams of two or three than for individuals. Conclusions: When performing quality assessments for systematic literature reviews, we recommend using three independent reviewers and adopting the median assessment. A quality checklist seems useful but it is difficult to ensure that the checklist is both appropriate and understood by reviewers. Furthermore, future experiments should ensure participants are given more time to understand the quality checklist and to evaluate the research papers.

show abstract

The Reliability of Peer Reviews of Papers on Information Systems

Cited by 22 publications

References 17 publications

Can we evaluate the quality of software engineering experiments?

Can we evaluate the quality of software engineering experiments?

Knowledge Market Design: A Field Experiment at Google Answers

Three empirical studies on the agreement of reviewers about the quality of software engineering experiments

Contact Info

Product

Resources

About