ICIS 2017 Panel Report: Break Your Shackles! Emancipating Information Systems from the Tyranny of Peer Review

Abstract

The paper presents the report of a panel that debated the review process in the information systems (IS) discipline at ICIS 2017 in Seoul, Korea. The panel asked the fundamental question of whether we need to rethink the way we review papers in the discipline. The panelists partnered with the audience to explore some reviewing limitations in IS today and the ways that reviewing in the discipline might change to address some of its difficulties. We first report key concerns with modern reviewing. We then present arguments for and against three proposals (i.e., paying for reviews, mandatory reviews, and open reviews) and a panel audience vote on the issues. We neither advocate for nor condemn these solutions but rather use them to illustrate what we believe represent the core underlying issues with reviewing in the IS discipline. Specifically, we believe the key stumbling blocks to effectively improving our review process include 1) a lack of empirical data on actual practice, 2) a lack of clear goals, and 3) an ignorance of the possible solutions to the review dilemma that the wider literature articulates.

Are we Concentrating on the Right Issues? A Response to the AIS Taskforce's Plan for the Future of IS Conferences

“…Following the AIS annual report of 2021, the total submissions for AMCIS, ECIS, ICIS, and PACIS combined was 3004 papers for the said reporting period (one can do the math of how many reviews and editorial reports were written). With the number of submissions increasing each year (due in part to the growth of the IS community), conference organizers and reviewers are being pushed to their limits (Chua et al, 2018). It is only through the tireless efforts of countless volunteers (to whom I am bound to great gratitude) that these events can even take place.…”

A Reflection on “Information Systems Research: Thinking Outside the Basket and Beyond the Journal”

“…It may also cause the journal to become risk adverse and to publish only papers that match standard expectations. Thus, I believe that ranking lists deprive other journals of these limited resources, which contributes to substantive problems with reviewing processes in the IS discipline (see Chua, Thatcher, Niederman, Chan, & Davidson, 2018) and impacts other important dimensions of research quality, such as timelines, variety, and creativity. This critique differs in substance from Fitzgerald et al's (2019) criticism of the reliability of the Senior Scholars basket list based on citation analysis, and the implications for assessing quality at the journal level differ as well.…”