Job Analysis and the Specification of Content for Licensure and Certification Examinations

In recent years, demand for performance assessments has continued to grow. However, performance assessments are notorious for lower reliability, and in particular, low reliability resulting from task specificity. Since reliability analyses typically treat the performance tasks as randomly sampled from an infinite universe of tasks, these estimates of reliability may not be accurate. For tests built according to a table of specifications, tasks are randomly sampled from different strata (content domains, skill areas, etc.). If these strata remain fixed in the test construction process, ignoring this stratification in the reliability analysis results in an underestimate of "parallel forms" reliability, and an overestimate of the person-by-task component. This research explores the effect of representing and misrepresenting the stratification appropriately in estimation of reliability and the standard error of measurement. Both multivariate and univariate generalizability studies are reported. Results indicate that the proper specification of the analytic design is essential in yielding the proper information both about the generalizability of the assessment and the standard error of measurement. Further, illustrative D studies present the effect under a variety of situations and test designs. Additional benefits of multivariate generalizability theory in test design and evaluation are also discussed.

show abstract

“…Kane 1997;Raymond 2001). The number of items assigned to each category is then based on the relative importance of those categories.…”

Section: Introductionmentioning

confidence: 99%

Using multivariate generalizability theory to assess the effect of content stratification on the reliability of a performance assessment

Keller

Clauser

Swanson

2010

Adv in Health Sci Educ

View full text Add to dashboard Cite

show abstract

“…Previous studies present various job analysis methods. Raymond (2001) argued that critical case methods, functional job analysis, position analysis survey, and capability requirement scales are used universally [7]. Furthermore, several analysis methods are identified, such as the initial analysis method, the confirmation comparison method, and the group debate method [8].…”

Section: Theoretical Backgroundmentioning

confidence: 99%

A Study on the Research Trend of Job Analysis in Korean Universities

Kim¹

2016

AJEMR

View full text Add to dashboard Cite

show abstract

“…Often, a job or practice analysis is undertaken to investigate the specific skills that are needed to perform adequately in the field. 25 Validity evidence based on response processes can take many forms. On an oral examination, for example, evidence needs to be gathered to show that the raters are using the evaluation criteria appropriately and are not being influenced by factors (e.g., sex, race, or training location) that are irrelevant to the intended interpretation of the scores.…”

Section: Validitymentioning

confidence: 99%

Review article: Assessment in anesthesiology education

Boulet

Murray

2011

Can J Anesth/J Can Anesth

View full text Add to dashboard Cite

Purpose The purpose of this article is to provide a review of the competencies and associated assessment techniques relevant to the practice of anesthesiology. Although many of the competencies are difficult to define and measure specifically, advances in assessment techniques have provided more opportunities to gather meaningful performance data. Principal findings Establishing the competence of anesthesiologists demands a host of measures, including standardized tests and less-structured peer evaluations. Simulation-based assessment will play an increasingly important role both in certification and in maintenance of certification for anesthesiologists. Conclusions While there are many psychometric challenges associated with the assessments pertinent to the education of anesthesiologists, technological advances combined with an increased awareness of sound measurement principles will yield more meaningful competency measures that can be used to improve the practice of anesthesiology.

show abstract

Job Analysis and the Specification of Content for Licensure and Certification Examinations

Cited by 85 publications

References 72 publications

Using multivariate generalizability theory to assess the effect of content stratification on the reliability of a performance assessment

Using multivariate generalizability theory to assess the effect of content stratification on the reliability of a performance assessment

A Study on the Research Trend of Job Analysis in Korean Universities

Review article: Assessment in anesthesiology education

Contact Info

Product

Resources

About