Combining Data on Criticality and Frequency in Developing Test Plans for Licensure and Certification Examinations

Abstract

Job analysis is a critical component in evaluating the validity of many high‐stakes testing programs, particularly those used for licensure or certification. The ratings of criticality and frequency of various activities that are derived from such job analyses can be combined in a number of ways. This paper develops a multiplicative model as a natural and effective way to combine ratings o f frequency and criticality in order to obtain estimates of the relative importance of different activities for practice. An example of the model's use is presented. The multiplicative model incorporates adjustments to ensure that the effective weights of frequency and criticality are appropriate.

An NCME Instructional Module on Developing and Administering Practice Analysis Questionnaires

“…The models can be specified to give more or less emphasis to different scales. For example, in the following additive model, criticality is weighted by a factor of three: Additive models are popular, but certain theoretical and statistical limitations have prompted some researchers to recommend the use of multiplicative models (Kane, Kingsbury, Colton, & Estes, 1989). Although multiplicative models are more complex, requiring that the rating data first be subjected to nonlinear transformations, the added complexity may prove worthwhile.…”

Knowledge and Skill Indicators Important for Beginning Teachers: A Survey of Pennsylvania Teachers and Teacher Educators

“…The use of a survey to either confirm or refute the judgments of committees is common practice when conducting job relevance studies (Kane, Kingsbury, Colton, & Estes, 1989;Tannenbaum & Rosenfeld, 1994;Knapp & Knapp, 1995;Tannenbaum, 1999). The use of a survey enables larger numbers of experts with potentially different but relevant points of view to be involved in the content validation process.…”

Obtaining Content Weights for Test Specifications From Job Analysis Task Surveys: An Application of the Many-Facets Rasch Model

“…Job analyses, also known as practice analyses, are used to provide a primary basis for defining test content and to validate licensure and certification examinations by providing a link between job tasks performed and test content specifications (AERA, APA, & NCME, 1999). Numerical methods exist for analyzing the survey data from multiple rating scales to obtain single composite task weights reflecting the relative importance of the tasks (Cascio & Ramos, 1986;Joseph & Taranath, 1999;Kane et al, 1989;Lunz, Stahl, & James, 1989;Nelson, 1993;Spray & Huang, 2000;Thomas & Kalohn, 1996;Wang, 2002). Still, issues such as the inappropriate treatment of the ordinal rating scales as interval measurement scales as well as the arbitrary use of subjective decisions to combine rating scales present psychometric concerns and open existing job analysis methods to criticism Joseph & Taranath, 1999;Lunz, Stahl, & James, 1989;Raymond, 2001;Spray & Huang, 2000;Wang, 2002Wang, , 2003Wang, , 2010Wang & Stahl, 2004.…”