Importance of Expert Judgment in Content-Related Validity Evidence

Abstract

The unified conceptualization of validity with regard to content-related evidence has been expressed succinctly by the authors of the Standards for Educational and Psychological Testing (AERA et al., 1985): Content-related evidence of validity is a central concern during [instrument] development, whether such development occurs in a research setting, in a publishing house, or in the context of professional practice. Expert professional judgment should play an integral part in developing the definition of what is to be measured, such as describing the universe of content, generating or selecting the content sample, and specifying the item format and scoring system. Thus, inferences about content are linked to [instrument] construction as well as to establishing evidence of validity after [an instrument] has been developed and chosen for use. (p. 11) This article has demonstrated the process of collecting content-related validity evidence in terms of the specific requirements of the Standards. Five standards were identified and interpreted according to the initial stages of instrument construction: domain specification, item development, and item, subscale, and scale content validation. The role of expert judgment during these stages and the variety of evidence that can be gathered were described. For most instruments, the review process would necessitate two meetings of 1 to 2 hours each to review the domain specifications and another two meetings to determine the match between the items and the specifications. The importance of these 8 hours or whatever additional time is needed to obtain the validity evidence was emphasized. Finally, an application of the Standards was provided to illustrate step-by-step how the judgmental review process can be planned and executed.

Development and psychometric evaluation of the radiographers’ competence scale

“…The number of items in all competence areas was increased except for the first; "organization and leadership". In line with the recommendations of Berk [22], four of the authors independently acted as experts when considering the logical consistency of the competencies and the number of items to be included in the RCS. Content and face validity were based on agreement between the four authors.…”

Reliability and Content Validity of a Comprehensive Discourse Completion Test of Interlanguage Pragmatics for Academic Situations

“…Taguchi (2006) argued that native speakers have got mastery over the situations and the communicative acts taking place. Berk (1990) argues that content-related evidence of validity is a central concern during "instrument" development and expert professional judgment should play an integral part in developing the test items and definition of what is to be measured. As expert judgment validity of the developed MCDT was proved, it can be concluded that the test enjoys a high content validity.…”

“…Secondly, findings revealed that the MDCT developed enjoyed high expert judgment validity based on content-related evidence. According to Berk (1990), content-related evidence of validity is a central concern during "instrument" development and expert professional judgment should play an integral part in developing the test items and definition of what is to be measured. The present study findings revealed a significant relationship between the rank orders of speech acts in the pragmatic test (MDCT), as the highest correlation was between apology and disagreement which was followed by the relationships between complaint and disagreement, request and refusal, request and apology, and refusal and apology.…”

“…Berk (1990) argues that content-related evidence of validity is a central concern during "instrument" development and expert professional judgment should play an integral part in developing the test items and definition of what is to be measured. A Multivariate Analysis of Variance (MANOVA) was run to compare speech acts of request, refusal, apology, complaint, and disagreement according to the native speakers' performance.…”

Evaluation of a Professional Development Model for Enhancing Knowledge, Skill and Confidence in Dysphagia Management

“…listing symptoms, labelling anatomical diagram, making diagnostic statements). To ensure appropriateness of test content, the items and the format of the test was checked and revised by a panel of three experienced University lecturers for content validity as recommended by Berk (1990). As the test questions were unchanged for each re-assessment, the authors accept that a learning effect may well have influenced performance on subsequent re-administration of the test (immediately post-training and at one month post-training).…”