Using Subject-Matter Experts to Assess Content Representation: An MDS Analysis

Abstract

Demonstration of content domain representation is of central importance in test validation. An expanded version of the method of content evaluation proposed by Sireci & Geisinger (1992) was evaluated with respect to a national licensure examination and a nationally standardized social studies achievement test. Two groups of 15 subject-matter experts (SMEs) rated the similarity of all item pairs comprising a test, and then rated the relevance of the items to the content domains listed in the test blueprints. The similarity ratings were analyzed using multidimensional scaling (MDS); the item relevance ratings were analyzed using procedures proposed by Hambleton (1984) and Aiken (1980). The SMES' perceptions of the underlying content structures of the tests emerged in the MDS solutions. All dimensions were germane to the content domains measured by the tests. Some of these dimensions were consistent with the content structure specified in the test blueprint, others were not. Correlation and regression analyses of the MDS item coordinates and item relevance ratings indicated that using both item similarity and item relevance data provided greater information of content representation than did using either approach alone. The implications of the procedure for test validity are discussed and suggestions for future research are provided.

La metodología Delphi como técnica de estudio de la validez de contenido

“…Aunque se han propuesto métodos de escalamiento multidimensional y análisis de conglomerados para tratar de solucionar este problema (Sireci y Geisinger, 1992, 1995, seguimos restringiendo el proceso a un mero análisis cuantitativo de la validez de contenido, que termina, como hemos señalado anteriormente, en la obtención de dichos indicadores y en la decisión de incluir o excluir el ítem del cuestionario.…”

What Is Going on During Vocabulary Lessons?

“…Sireci and Geisinger (1995) summarized the benefits for using MDS in establishing the construct validity of an assessment. The rationale being that an ordination or the tendency of like items to be similar in proximity and unlike items to be further away from each other provides evidence that the instrument is indeed discriminating material-type between defined construct dimensions, in this case "control" and "number of senses."…”

Development of an Early Hebrew Oral Reading Fluency Measure

“…As is commonly done to establish evidence for content validity (e.g. Sireci & Geisinger, 1995), the Hebrew reading measures were presented to Subject Matter Experts (SMEs). There were two types of SMEs, seasoned Hebrew language instructors and elementary school Hebrew teachers from schools of diverse denominations.…”