Peer review is a gatekeeper, the final arbiter of what is valued in academia, but it has been criticized in relation to traditional psychological research criteria of reliability, validity, generalizability, and potential biases. Despite a considerable literature, there is surprisingly little sound peer-review research examining these criteria or strategies for improving the process. This article summarizes the authors' research program with the Australian Research Council, which receives thousands of grant proposals from the social science, humanities, and science disciplines and reviews by assessors from all over the world. Using multilevel cross-classified models, the authors critically evaluated peer reviews of grant applications and potential biases associated with applicants, assessors, and their interaction (e.g., age, gender, university, academic rank, research team composition, nationality, experience). Peer reviews lacked reliability, but the only major systematic bias found involved the inflated, unreliable, and invalid ratings of assessors nominated by the applicants themselves. The authors propose a new approach, the reader system, which they evaluated with psychology and education grant proposals and found to be substantially more reliable and strategically advantageous than traditional peer reviews of grant applications.

Background and Purpose. Arm lymphedema following breast cancer surgery is a continuing problem. In this study, we assessed the reliability and validity of circumferential measurements and water displacement for measuring upper-limb volume. Subjects. Participants included subjects who had had breast cancer surgery, including axillary dissection—19 with and 22 without a diagnosis of arm lymphedema—and 25 control subjects. Methods. Two raters measured each subject by using circumferential tape measurements at specified distances from the fingertips and in relation to anatomic landmarks and by using water displacement. Interrater reliability was calculated by analysis of variance and multilevel modeling. Volumes from circumferential measurements were compared with those from water displacement by use of means and correlation coefficients, respectively. The standard error of measurement, minimum detectable change (MDC), and limits of agreement (LOA) for volumes also were calculated. Results. Arm volumes obtained with these methods had high reliability. Compared with volumes from water displacement, volumes from circumferential measurements had high validity, although these volumes were slightly larger. Expected differences between subjects with and without clinical lymphedema following breast cancer were found. The MDC of volumes or the error associated with a single measure for data based on anatomic landmarks was lower than that based on distance from fingertips. The mean LOA with water displacement were lower for data based on anatomic landmarks than for data based on distance from fingertips. Discussion and Conclusion. Volumes calculated from anatomic landmarks are reliable, valid, and more accurate than those obtained from circumferential measurements based on distance from fingertips. [Taylor R, Jayasinghe UW, Koelmeyer L, et al. Reliability and validity of arm volume measurements for assessment of lymphedema. Phys Ther. 2006;86:205–214.]

The peer review of grant proposals is very important to academics from all disciplines. Although there is limited research on the reliability of assessments for grant proposals, previously reported single-rater reliabilities have been disappointingly low (between 0.17 and 0.37). We found that the single-rater reliability of the overall assessor rating for Australian Research Council grants was 0.21 for social science and humanities (2870 ratings, 1928 assessors and 687 proposals) and 0.19 for science (7153 ratings, 4295 assessors and 1644 proposals). We used a multilevel, cross-classification approach (level 1, assessor and proposal cross-classification; level 2, field of study), taking into account that 34% of the assessors evaluated more than one proposal. Researcher-nominated assessors (those chosen by the authors of the research proposal) gave higher ratings than panel-nominated assessors chosen by the Australian Research Council, and proposals from more prestigious universities received higher ratings. In the social sciences and humanities, the status of Australian universities had significantly more effect on Australian assessors than on overseas assessors. In science, ratings were higher when assessors rated fewer proposals and apparently had a more limited frame of reference for making such ratings and when researchers were professors rather than non-professors. Particularly, the methodology of this large scale study is applicable to other forms of peer review (publications, job interviews, awarding of prizes and election to prestigious societies) where peer review is employed as a selection process. Copyright 2003 Royal Statistical Society.

BackgroundLimited evidence exists regarding the relationship between health literacy and health-related quality of life (HRQoL) in Australian patients from primary care. The objective of this study was to investigate the impact of health literacy on HRQoL in a large sample of patients without known vascular disease or diabetes and to examine whether the difference in HRQoL between low and high health literacy groups was clinically significant.MethodsThis was a cross-sectional study of baseline data from a cluster randomised trial. The study included 739 patients from 30 general practices across four Australian states conducted in 2012 and 2013 using the standard Short Form Health Survey (SF-12) version 2. SF-12 physical component score (PCS-12) and mental component score (MCS-12) are derived using the standard US algorithm. Health literacy was measured using the Health Literacy Management Scale (HeLMS). Multilevel regression analysis (patients at level 1 and general practices at level 2) was applied to relate PCS-12 and MCS-12 to patient reported life style risk behaviours including health literacy and demographic factors.ResultsLow health literacy patients were more likely to be smokers (12 % vs 6 %, P = 0.005), do insufficient physical activity (63 % vs 47 %, P < 0.001), be overweight (68 % vs 52 %, P < 0.001), and have lower physical health and lower mental health with large clinically significant effect sizes of 0.56 (B (regression coefficient) = −5.4, P < 0.001) and 0.78(B = -6.4, P < 0.001) respectively after adjustment for confounding factors. Patients with insufficient physical activity were likely to have a lower physical health score (effect size = 0.42, B = −3.1, P < 0.001) and lower mental health (effect size = 0.37, B = −2.6, P < 0.001). Being overweight tended to be related to a lower PCS-12 (effect size = 0.41, B = −1.8, P < 0.05). Less well-educated, unemployed and smoking patients with low health literacy reported worse physical health. Health literacy accounted for 45 and 70 % of the total between patient variance explained in PCS-12 and MCS-12 respectively.ConclusionsAddressing health literacy related barriers to preventive care may help reduce some of the disparities in HRQoL. Recognising and tailoring health related communication to those with low health literacy may improve health outcomes including HRQoL in general practice.