Mapping the Landscape of Peer Review in Computing Education Research

Abstract

Peer review is a mainstay of academic publication -indeed, it is the peer-review process that provides much of the publications' credibility. As the number of computing education conferences and the number of submissions increase, the need for reviewers grows. This report does not attempt to set standards for reviewing; rather, as a first step toward meeting the need for well qualified reviewers, it presents an overview of the ways peer review is used in various venues, both inside computing education and, for comparison, in closely-related areas outside our field. It considers four key components of peer review in some depth: criteria, the review process, roles and responsibilities, and ethics and etiquette. To do so, it draws on relevant literature, guidance and forms associated with peer review, interviews with journal editors and conference chairs, and a limited survey of the computing education research community. In addition to providing an overview of practice, this report identifies a number of themes running through the discourse that have relevance for decision making about how best to conduct peer review for a given venue.

Assessing the Quality of Student-Generated Content at Scale: A Comparative Analysis of Peer-Review Models

“…A common solution is to rely on the wisdom of the crowd rather than one person by employing a redundancy-based strategy and assigning the same task to multiple users. This solution has been also utilised in other types of assessments such as peer reviewing of academic manuscripts [18] or grant proposals [19] and more widely by the crowdsourcing community [20]. However, it raises a new problem commonly referred to as the consensus problem: in the absence of ground truth, how can we optimally integrate the decisions made by multiple people towards an accurate final decision?…”