On the Reporting of Measurement Uncertainty and Reliability for U.S. Educational and Licensure Tests: Report to Ofqual

Phelps, Richard P.

doi:10.2139/ssrn.2687370

Cited by 5 publications

(12 citation statements)

References 5 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…One of the areas where regulation may play a role that is being explored by Ofqual is how to improve public understanding of reliability and increase public confidence in the national examinations system in England. It is difficult for policy-makers, regulators and assessment providers to communicate with the public about this technically complex and intrinsically bad-news story, especially in a context of frequent media stories about the allegedly inadequate examinations system (Boyle, Opposs, & Kinsella, 2009;Newton, 2005; Office of Qualifications and Examinations Regulation [Ofqual], 2009;Phelps, Zenisky, Hambleton, & Sireci, 2010). However, such difficulties are not unique; physicians may need to gain informed consent even from children or people with learning difficulties (Malhotra & Subodh, 2009) or researchers may seek to inform the public about the uses of GM crops in a context of media hysteria (Johnson, 2004).…”

Section: Discussionmentioning

confidence: 99%

“…Such negativity underlines the difficulties inherent in such activity. However, the difficulty of communicating about reliability does not diminish the obligation to do it (Newton, 2005;Phelps et al, 2010). Gaining insight into public perceptions of reliability based on sound research, then using insights from such research to have practical impact on public policy, seems to be the correct way to proceed.…”

Section: Discussionmentioning

confidence: 99%

See 1 more Smart Citation

Public perceptions of reliability in examination results in England

He¹,

Boyle

Opposs³

2011

Evaluation & Research in Education

View full text Add to dashboard Cite

Building on findings from existing qualitative research into public perceptions of reliability in examination results in England, a questionnaire was developed and administered to samples of teachers, students and employers to study their awareness of and opinions about various aspects of reliability quantitatively. Main findings from the study include: there was substantial variability in the understanding of reliability concepts and attitudes to unreliability in examination results among the respondents; the majority of the respondents appeared to understand the assessment process and the factors that affect students' performances in examinations; to a degree, the respondents also understood the factors that could introduce inconsistency in examination results; the respondents showed various degrees of experience of the examination process and acceptance of measurement error in examination results; the level of tolerance of the respondents for assessment error to some degree was positively correlated to the level of belief about the examinations system, knowledge of aspects of unreliability and approaches to trust. Based on findings from this study and those from other studies, steps that should be taken to communicate with the public about reliability of assessment outcomes have been proposed.

show abstract

Section: Discussionmentioning

confidence: 99%

Section: Discussionmentioning

confidence: 99%

Public perceptions of reliability in examination results in England

He¹,

Boyle

Opposs³

2011

Evaluation & Research in Education

View full text Add to dashboard Cite

show abstract

“…This is addressed in Hattie (2010) who found that teachers responded best to pictorial representations of standard error rather than explanations in words. Phelps et al (2010) also discuss this issue in a paper which looked specifically at reporting of measurement uncertainty in the USA. This covered both how it is reported and made available in technical manuals and how it is included in individual results reports.…”

Section: Discussionmentioning

confidence: 99%

“…This may be presented as a graph or as a range of scores, or in some cases both. As described by Goodman and Hambleton (2004) and Phelps et al (2010), this is a practice in some US state assessments. It is also used in some Canadian provincial school assessments.…”

Section: Reporting Of Error and Uncertaintymentioning

confidence: 99%

See 1 more Smart Citation

Reporting error and reliability to test-takers: an international review

Bradshaw¹,

Wheater²

2013

Research Papers in Education

View full text Add to dashboard Cite

This review examined a range of approaches internationally to the reporting of assessment results for individual students, with a particular focus on how results are represented, the level of detail reported and the steps taken to quantify, report and explain error and uncertainty in the results' reports or certificates given to students in a range of countries. The review focused specifically on high-stakes assessments taken towards the end of schooling with the purpose of providing certification for employment or further study. Outside the USA, there were few examples of the inclusion of information on error or uncertainty in students' individual results. The paper discusses the possibilities for including such information in a way which will be understood by a nontechnical audience and will not be misinterpreted by test users. The balance required between ensuring that there are high standards of information for individuals and the dangers of damaging public confidence in the assessment system is considered. IntroductionThis research was part of a programme of research into the reliability of results of qualifications, examinations and assessments in England on behalf of Ofqual, the examinations regulator. As part of this reliability programme, an international review of approaches to reporting results was undertaken. This review aimed to examine the range of approaches taken to the reporting of assessment results for individual students, with a particular focus on how results are represented, the level of detail reported and the steps taken to quantify, report and explain error and uncertainty in the results reports or certificates given to students. The review focused specifically on high-stakes assessments taken towards the end of schooling with the purpose of providing certification for employment or further study (Bradshaw and Wheater 2009). The final report included a detailed descriptive classification of various aspects of these assessments. This paper discusses one aspect of this classification: the reporting of error or uncertainty of results. This may be error at an individual level or at a general, group or cohort level. Consideration of validity and other reliability issues, such as how reported estimates of reliability or error have been derived, were outside the scope of the study. Further detail on the study can be found in Bradshaw and Wheater (2009).

show abstract

Comparing the influence of various measurement error presentations in test score reports on educational decision-making

Otter

Muilenburg

Wools

et al. 2018

Assessment in Education: Principles, Policy & Practice

View full text Add to dashboard Cite

a Faculty of Behavioral, management and social sciences (Bms), university of twente, enschede, the netherlands; b citolab, cito institute for educational measurement, Arnhem, the netherlands; c Psychometric research centre, cito institute for educational measurement, Arnhem, the netherlands ABSTRACT This study investigated (1) the extent to which presentations of measurement error in score reports influence teachers' decisions and (2) teachers' preferences in relation to these presentations. Three presentation formats of measurement error (blur, colour value and error bar) were compared to a presentation format that omitted measurement error. The results from a factorial survey analysis showed that the position of a score in relation to a cut-off score impacted most significantly on decisions. Moreover, the teachers (N = 337) indicated the need for additional information significantly more often when the score reports included an error bar compared to when they omitted measurement error. The error bar was also the most preferred presentation format. The results were supported in thinkaloud protocols and focus groups, although several interpretation problems and misconceptions of measurement error were identified.

show abstract

On the Reporting of Measurement Uncertainty and Reliability for U.S. Educational and Licensure Tests: Report to Ofqual

Cited by 5 publications

References 5 publications

Public perceptions of reliability in examination results in England

Public perceptions of reliability in examination results in England

Reporting error and reliability to test-takers: an international review

Comparing the influence of various measurement error presentations in test score reports on educational decision-making

Contact Info

Product

Resources

About