Measure Twice, Cut down Error: A Process for Enhancing the Validity of Survey Scales

Abstract

For years psychologists across many subfields have undertaken the formidable challenge of designing survey scales to assess attitudes, opinions, and behaviors. Correspondingly, scholars have written much to guide researchers in this undertaking. Yet, many new scales violate established best practices in survey design, suggesting the need for a new approach to designing surveys. This article presents 6 steps to facilitate the construction of questionnaire scales. Unlike previous processes, this one front loads input from other academics and potential respondents in the item-development and revision phase with the goal of achieving credibility across both populations. Specifically, the article describes how (a) a literature review and (b) focus group-interview data can be (c) synthesized into a comprehensive list to facilitate (d) the development of items. Next, survey designers can subject the items to (e) an expert review and (f) cognitive pretesting before executing a pilot test.

Pathways to Educational Success Among Refugees: Connecting Locally and Globally Situated Resources

“…); most of the questions were closed, a few were open-response. We developed the survey following the process outlined by Gehlbach and Brinkworth (2011), including a comprehensive literature review, consultation with Somali university students, and two rounds of cognitive pre-testing.…”

The role of the associate dean in UK universities: distributed leadership in action?

“…The questionnaire was developed following guidelines put forward by Gehlbach and Brinkworth (2011) and subject to an initial peer review followed by a full piloting exercise involving associate deans in the two researchers' universities. Subsequently, an invitation and link to the survey were sent out via email to 472 associate deans across the UK (England, Scotland, and Wales only).…”

Beyond Special and General Education as Identity Markers: The Development and Validation of an Instrument to Measure Preservice Teachers’ Understanding of The Effects of Intersecting Sociocultural Identities

“…For each of these three categories, the researcher explores how constructs identified in existing measures (e.g., attitudes toward inclusion, cultural competence, multicultural competence, and teaching for social justice) relate to intersectional competence. As Gehlbach and Brinkworth (2011) explained, knowledge of the literature helps designers of instruments to "define their construct so as to situate it within, connect it to, and differentiate it from related concepts" (p. 2). The researcher will examine the "degree of overlap between" intersectional competence and "related, but distinct constructs" (p. 2).…”

“…The extant literature of scholars who incorporate intersectionality theory to special education research is scarce but growing. In accordance to the best practices that have been delineated for scale and survey development (Gehlbach & Brinkworth, 2011;Simms, 2008), the process of designing an instrument to assess a construct (in the case of this study, intersectional competence) begins with a thorough review of the literature. The review of intersectionality and special education research yielded the following three indicators of intersectional competence: (a) the ability to clearly identify sociocultural group categories (e.g., Arms, Bickett, & Graf , 2008;Artiles, 2013) and markers of difference, such as the markers of diversity that are identified in the NCATE (2008) standards; (b) an emphasis on the interlocking and simultaneous effects of multiple markers of difference (e.g., Ferri & Connor, 2008;García & Ortiz, 2013;Natapoff, 2005); and (c) an understanding of the systems of oppression and marginalization (e.g., Connor, 2006;Grant & Zwier, 2011;McCall & Skrtic) that occur at the intersection of multiple markers of difference, with special attention to the intersection of disability with other markers of difference (Connor, 2008).…”

“…Gehlbach and Brinkworth (2011) proposed a six-step process for enhancing the validity of survey scales that first includes collection and analysis of qualitative data and then moves on to quantitative analysis. Although the authors of this six-step approach do not call their sequence a mixed method design, they acknowledge that the process: …(a) uses a broader range of techniques, (b) encourages scholars to be more collaborative with other researchers and with potential respondents during item development, and (c) increases the emphasis on validity early in the process should ultimately produce more efficient, valid scales while requiring fewer pilot tests.…”