A Consumer’s Guide to Setting Performance Standards on Criterion-Referenced Tests

Berk, Ronald A.

doi:10.3102/00346543056001137

Cited by 178 publications

(107 citation statements)

References 43 publications

Supporting

Mentioning

105

Contrasting

Unclassified

Order By: Relevance

“…Standard setting has been extensively reviewed (Hambleton and Powell, 1983;Berk, 1986;Cusimano, 1996). In medical education absolute methods for standard setting assessments have been developed for large scale licensing or certification examinations, and much of the reported research derives from their use in this context (Downing et al, 2003).…”

Section: Introductionmentioning

confidence: 99%

“…The modified Angoff method (Case and Swanson, 1996) was selected for SAQ-based papers. This method is readily adaptable to such questions, is well documented as a suitable method for high stakes assessments (Berk, 1986) and has been recommended for its simplicity, ease of application and psychometric characteristics (Norcini et al, 1987). The Ebel method (Ebel, 1979) was chosen for EMQ-based papers, as this alternative test-centred method can provide information on item difficulty and relevance, thus offering the potential to categorise items for future use.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Estimating the Minimum Number of Judges Required for Test-centred Standard Setting on Written Assessments. Do Discussion and Iteration have an Influence?

Fowell

Fewtrell

2006

Adv in Health Sci Educ

View full text Add to dashboard Cite

Absolute standard setting procedures are recommended for assessment in medical education. Absolute, test-centred standard setting procedures were introduced for written assessments in the Liverpool MBChB in 2001. The modified Angoff and Ebel methods have been used for short answer question-based and extended matching question-based papers, respectively. Data collected has been analysed to investigate whether reliable standards can be achieved for small-scale, medical school-based assessments, to establish the minimum number of judges required and the effect of a discussion phase on reliability. The root mean squared error (RMSE) has been used as a measure of reliability and used to compute 95% confidence intervals for comparison to the examination statistics. The RMSE has been used to calculate the minimum number of judges required to obtain a predetermined minimum level of reliability, and the effect of the number of judges and number of items have been examined. Values of the RMSE obtained vary from 0.9 to 2.2%. Using average variances across each paper type, the minimum number of judges to obtain a RMSE of less than 2% is 10 or more judges before discussion or 6 or more judges after discussion. The results indicate that including a discussion phase improves the reliability and reduces the minimum number of judges required. Decision studies indicate that increasing the number of questions included in the assessments would not significantly improve the reliability of the standard setting.

show abstract

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Estimating the Minimum Number of Judges Required for Test-centred Standard Setting on Written Assessments. Do Discussion and Iteration have an Influence?

Fowell

Fewtrell

2006

Adv in Health Sci Educ

View full text Add to dashboard Cite

show abstract

“…The reliability of the knowledge items was 0.92 as determined by Livingston's coefficient for criterion-referenced tests using a conservative passing score of 70 %, indicating a high degree of reliability (Table 1) [31][32][33][34][35][36]. Likewise, Cronbach's alpha for continuous items were acceptable for all scales.…”

Section: Resultsmentioning

confidence: 99%

Evaluation of a Childhood Obesity Prevention Online Training Certificate Program for Community Family Educators

et al. 2016

View full text Add to dashboard Cite

Community family educators have the opportunity to incorporate childhood obesity prevention concepts in their programming with families of young children, but often lack formal health and nutrition education. The purpose of this feasibility study was to create an online training certificate program for community family educators and assess the program's effectiveness at improving participant's knowledge, attitudes, and intended and actual behaviors related to healthy lifestyles. Community family educators (n = 68) completed an online pretest, viewed 13 brief videos (8-15 min) focused on childhood obesity related topics and took mini-knowledge self-checks after each video followed by an online posttest. At posttest, paired t tests showed participants' childhood obesity prevention related knowledge (i.e., nutrition, physical activity, screen time and sleep) improved significantly (p < 0.001). Participants' attitudes toward parenting behaviors related to feeding practices, family meals, physical activity, screen time control and parent modeling significantly (p < 0.05) improved. Improvements also were seen in participants' intentions to promote obesity prevention behaviors (i.e., age appropriate portions sizes, adequate physically active, and parental role modeling). Furthermore, changes in personal health behaviors at posttest revealed participants had significantly (p < 0.05) greater dietary restraint, improvements in sleep quality, and reductions of use of electronic devices during meals and snacks. Overall, participants were very satisfied with the training program, felt comfortable with skills acquired, and enjoyed the program. Findings suggest this online training program is a feasible and effective method for improving community family educators' knowledge, attitudes, and intentions for obesity-prevention related parenting practices.

show abstract

“…We have left the matter of decision rules until the end because standard setters need all the evidence--from reliability data to methods of equating--before setting standards. A number of methods have been proposed (Berk, 1986;Klein, 1986;Livingston and Kieky, 1982;Meskauskas and Norcini, 1981;Popham, 1987), most of which involve having a group of experts make judgments about passing scores based on test items, test scores, and field test data.…”

Section: Decision Rules and Standard Settingmentioning

confidence: 99%

The validity of performance assessments for teacher licensure: Connecticut's ongoing research

Pecheone

Carey

1990

J Pers Eval Educ

View full text Add to dashboard Cite

A Consumer’s Guide to Setting Performance Standards on Criterion-Referenced Tests

Cited by 178 publications

References 43 publications

Estimating the Minimum Number of Judges Required for Test-centred Standard Setting on Written Assessments. Do Discussion and Iteration have an Influence?

Estimating the Minimum Number of Judges Required for Test-centred Standard Setting on Written Assessments. Do Discussion and Iteration have an Influence?

Evaluation of a Childhood Obesity Prevention Online Training Certificate Program for Community Family Educators

The validity of performance assessments for teacher licensure: Connecticut's ongoing research

Contact Info

Product

Resources

About