Obtaining Test Blueprint Weights From Job Analysis Surveys

Abstract

A method for combining multiple scale responses from job or task surveys based on a hierarchical ranking scheme is presented. A rationale for placing the resulting ordinal information onto an interval scale of measurement using the Rasch Rating Scale Model is also provided. After a simple linear transformation, the item or task parameter estimates can be used to obtain item weights to be used in constructing test blueprints. Prior weights can then be used to modify the item weights after data collection, based either on content balancing requirements or Bayesian prior content weights from SMEs (subject matter experts). Finally a method is suggested to link two or more surveys, again using the Rasch Rating Scale Model and the computer program, Bigsteps, when it is desirable to shorten the length of the typical job or task survey.

Obtaining Content Weights for Test Specifications From Job Analysis Task Surveys: An Application of the Many-Facets Rasch Model

“…The scale could be 1 to 4 or the categories could even be numbered 0, 2, 4, 6 without changing the ordinal meaning of the scale. One criticism of some existing job analysis methods is the inappropriate treatment of this type of ordinal scale as interval measurement scales (Lunz et al, 1989;Spray & Huang, 2000). Simply adding up or multiplying the ordinal rating data across tasks and across scales, and using the summation or product as an interval measurement of the task importance is a common problem for a number of job analysis methods (Joseph & Taranath, 1999).…”

“…Job analyses, also known as practice analyses, are used to provide a primary basis for defining test content and to validate licensure and certification examinations by providing a link between job tasks performed and test content specifications (AERA, APA, & NCME, 1999). Numerical methods exist for analyzing the survey data from multiple rating scales to obtain single composite task weights reflecting the relative importance of the tasks (Cascio & Ramos, 1986;Joseph & Taranath, 1999;Kane et al, 1989;Lunz, Stahl, & James, 1989;Nelson, 1993;Spray & Huang, 2000;Thomas & Kalohn, 1996;Wang, 2002). Still, issues such as the inappropriate treatment of the ordinal rating scales as interval measurement scales as well as the arbitrary use of subjective decisions to combine rating scales present psychometric concerns and open existing job analysis methods to criticism Joseph & Taranath, 1999;Lunz, Stahl, & James, 1989;Raymond, 2001;Spray & Huang, 2000;Wang, 2002Wang, , 2003Wang, , 2010Wang & Stahl, 2004.…”

“…To remedy the problem of mistreating the ordinal scales, several methods using Rasch models have been proposed to scale ordinal job analysis rating data into a linear interval measurement (Lunz et al, 1989;Spray & Huang, 2000). Lunz and colleagues introduced the first application of the Rasch Rating Scale Model (RRSM) (Andrich, 1988) for analyzing job analysis task survey data.…”

“…According to the Standards for Educational and Psychological Testing (American Educational Research Association [AERA], American Psychological Association [APA], & National Council on Measurement in Education [NCME], 1999), validation of these examinations depends mainly on contentrelated evidence, with job analysis (or practice analysis) providing the primary basis for defining the test content domain. To define test content domains and develop test specifications for licensure and certification examinations, the most common approach of a job analysis is to use a task inventory questionnaire (Newman, Slaughter, & Taranath, 1999;Raymond, 2001Raymond, , 2002Spray & Huang, 2000). There is a common core of procedures typically employed by a task inventory job analysis (Wang, 2010;Wang, Schnike, & Witt, 2005):…”

“…Results from the Many-Facets Rasch Model are compared with those calculated from the Rasch Rating Scale Model (RRSM) (Spray & Huang, 2000) using two examples of actual job analysis data from diverse professions. In addition, this article proposes a solution for establishing the origin of the percentage scale when transferring the task importance weights from a logit unit into percentage weights.…”