A Practical Guide to Practice Analysis for Credentialing Examinations

Abstract

Practice analysis (i.e., job analysis) serves as the cornerstone for the development of credentialing examinations and is generally used as the primary source of evidence when validating scores on such exams. Numerous methodological questions arise when planning and conducting a practice analysis, but there is little consensus in the measurement community regarding the answers to these questions. This article offers recommendations concerning the following issues: selecting a method of practice analysis; developing rating scales to describe practice; determining the content of test plans; using multivariate procedures for structuring test plans; and determining topic weights for test plans. The article closes by suggesting several references for further reading.

Practice Indicators of Suboptimal Care and Avoidable Adverse Events

“…Yet, we need to conduct a more comprehensive practice analysis to fully capture that domain, including, for example, patient safety problems and medical errors in the community, home, and long-term care. Depending on the candidates to assess and the practice analysis procedures used for criterionreferenced licensing examinations, 46,47 we must decide what omissions and commissions occur frequently enough in practice and with sufficient health consequences to be considered essential competencies for a national exam intended for supervised or unsupervised practice. We then can incorporate such analyses and decisions into a broader, more comprehensive practice model for blueprinting that can guide the content selection for the entire exam.…”

An NCME Instructional Module on Developing and Administering Practice Analysis Questionnaires

“…Cognitively complex judgments are necessary for some practice analysis studies; but collecting these judgments may impact sampling, scale design, and data collection strategies. Although typical practitioners are the best source of information for ratings of task frequency or task difficulty, judgments about task criticality might better be left to seasoned SMEs (Kane, 1997; Raymond, 2002a). A questionnaire is not always the most effective method for collecting job‐related data.…”

Obtaining Content Weights for Test Specifications From Job Analysis Task Surveys: An Application of the Many-Facets Rasch Model

“…According to the Standards for Educational and Psychological Testing (American Educational Research Association [AERA], American Psychological Association [APA], & National Council on Measurement in Education [NCME], 1999), validation of these examinations depends mainly on contentrelated evidence, with job analysis (or practice analysis) providing the primary basis for defining the test content domain. To define test content domains and develop test specifications for licensure and certification examinations, the most common approach of a job analysis is to use a task inventory questionnaire (Newman, Slaughter, & Taranath, 1999;Raymond, 2001Raymond, , 2002Spray & Huang, 2000). There is a common core of procedures typically employed by a task inventory job analysis (Wang, 2010;Wang, Schnike, & Witt, 2005):…”