Reviewing the reviewers: A study of author perception on peer reviews in computer science

Kühne, Conny; Böhm, Klemens; Yue, Jing Zhi

doi:10.4108/icst.collaboratecom.2010.33

Cited by 5 publications

(12 citation statements)

References 14 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…PR were positive (see Fig. 1), the explanation may be in line with findings from Kuhne et al (2010) and Zhang, Zhang, and Law (2012); authors like reviews that like their submissions. Keeping in mind the original items of F1 and F4 (enhancements, true recognition of strengths, new ideas for discussion, complementary aspects, new references, technical clarifications, mistake removals), the empirical data support notions of appreciative inquiry (Patton, 2011), AfL (Black, 2015;Wiliam, 2011), and feedforward orientation (Cathcart, Greer, & Neale, 2014).…”

Section: Figuresupporting

confidence: 78%

“…), the explanation may be in line with findings from Kuhne et al . () and Zhang, Zhang, and Law (); authors like reviews that like their submissions.…”

Section: Resultsmentioning

confidence: 99%

“…Feedback may be very dispiriting for the authors if they see that the lack of reviewer expertise has led to totally irrelevant suggestions or directions (Kuhne, Böhm, & You, ). Ware () concludes, from several original studies, that reviewers’ opinions often differ from each other, that is, inter‐rater reliability is weak.…”

Section: Pr As Critical Appraisal From the Developmental Viewpointmentioning

confidence: 99%

See 2 more Smart Citations

Peer review in the development of academic articles: Experiences of Finnish authors in the educational sciences

Atjonen

2018

Learned Publishing

View full text Add to dashboard Cite

This research examined author experiences of the developmental feedback regarding peer review of journal articles. It focused on quite experienced authors in the educational sector of all eight universities in Finland. Data were gathered using an electronic questionnaire, which was completed by 121 respondents. The majority of authors perceived reviewer feedback as developmental and appreciated critical but constructive review. Over 60% of respondents found reviewers usually keen to improve the article and to provide pertinent critique. Less than 8% reported frequent experience of reviewers who were not experts or provided unjustified accusations. A third reported reviewer feedback that was frequently biased or based on different paradigms. The most published authors were the most satisfied with the developmental feedback, but postdoctoral researchers had received more developmental complementary feedback than any other groups. The results indicate that the peer review system remains useful for ensuring article quality, but some attention is still required to ensure developmental feedback that will help the authors enhance their manuscripts.

show abstract

Section: Figuresupporting

confidence: 78%

“…), the explanation may be in line with findings from Kuhne et al . () and Zhang, Zhang, and Law (); authors like reviews that like their submissions.…”

Section: Resultsmentioning

confidence: 99%

Section: Pr As Critical Appraisal From the Developmental Viewpointmentioning

confidence: 99%

See 1 more Smart Citation

Peer review in the development of academic articles: Experiences of Finnish authors in the educational sciences

Atjonen

2018

Learned Publishing

View full text Add to dashboard Cite

show abstract

“…As we review the standard guidelines 3,4,5,6 for peer-reviewing in machine learning (ML) and natural language processing (NLP) conferences, we learn that the community expects a good review that covers more sections and aspects of the reviewed manuscript (Gregory and Denniss, 2019; Kühne et al, 2010). Having this motivation led us to develop a justifiable informativeness score which enables editors to automatically identify good reviews and isolate those that are less thorough.…”

Section: Proposed Methodsmentioning

confidence: 99%

“…The peer review process is the central mechanism for validating scientific research (Siler et al, 2015). A good review typically provides feedback on one or more sections and aspects while reviewing the manuscript/paper 1 , rather than just one section, say the Introduction (Kühne et al, 2010). Therefore, reviews covering more sections and aspects are more likely helpful to the author.…”

Section: Introductionmentioning

confidence: 99%

BetterPR: A Dataset for Estimating the Constructiveness of Peer Review Comments

Bharti

Ghosal

Agarwal

et al. 2022

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Peer reviews are intended to give authors constructive and informative feedback. It is expected that the reviewers will make constructive suggestions over certain aspects, e.g., novelty, clarity, empirical and theoretical soundness, etc., and sections, e.g., problem definition/idea, datasets, methodology, experiments, results, etc., of the paper in a detailed manner. With this objective, we analyze the reviewer's attitude toward the work. Aspects of the review are essential to determine how much weight the editor/chair should place on the review in making a decision. In this paper, we used a publicly available Peer Review Analyze dataset of peer review texts manually annotated at the sentence level (13.22 k sentences) across two layers: Paper Section Correspondence and Paper Aspect Category. We transform these categorical annotations to derive an informativeness score of the review based on the review's coverage across section correspondence, aspects of the paper, and reviewer-centric uncertainty associated with the review. We hope that our proposed methods, which are motivated towards automatically estimating the quality of peer reviews in the form of informativeness scores, will give editors an additional layer of confidence for the automatic judgment of review quality. We make our codes available at https: //github.com/PrabhatkrBharti/ informativeness.git.

show abstract

Ethics in Peer Review of Academic Journal Articles as Perceived by Authors in the Educational Sciences

Atjonen

2018

J Acad Ethics

View full text Add to dashboard Cite

Reviewing the reviewers: A study of author perception on peer reviews in computer science

Cited by 5 publications

References 14 publications

Peer review in the development of academic articles: Experiences of Finnish authors in the educational sciences

Peer review in the development of academic articles: Experiences of Finnish authors in the educational sciences

BetterPR: A Dataset for Estimating the Constructiveness of Peer Review Comments

Ethics in Peer Review of Academic Journal Articles as Perceived by Authors in the Educational Sciences

Contact Info

Product

Resources

About