Measuring how college affects students: Social desirability and other potential biases in college student self‐reported gains

Abstract

Colleges and universities are increasingly using national surveys to assess their students' learning and development. Given the importance of the first year of college for student adjustment and retention (Tinto, 1993), some of these surveys are designed specifically to gauge the experiences and outcomes of first-year students. These large-scale surveys provide valuable information for higher education researchers, administrators, and practitioners. However, a growing body of research has questioned the validity of the measures of student self-reported gains in learning and development that are frequently employed in these assessments (Bowman,

The Measurement of Socially Responsible Leadership: Considerations in Establishing Psychometric Rigor

“…Debriefing explored problem questions focusing on how they could be improved for clarity, comprehension, and ease of responding accurately or whether they should be removed. Finally, similar to findings related to socially desirous responding, the halo effect appears to be more pronounced with samples of students early in their college careers (Pike, 1999;Bowman & Hill, 2011). Using the SRLS specifically with samples of college seniors may ameliorate further concerns related to the halo effect in cross-sectional studies.…”

“…Studies demonstrate the ways in which social desirability skews results (Bowman & Hill, 2011;Porter, 2011) validating Porter's (2011 concern that «social desirability on college student surveys may lead to distorted and misleading conclusions» (p. 74). Consistent with recommendations (Gonyea, 2005;Porter, 2011), Tyree (1998) embedded the Marlowe-Crowne Scale of Social Desirability (Crowne & Marlowe, 1960) in her original research that led to the creation of the SRLS.…”

“…Results again yielded no concerning relationships between items on the SRLS and socially desirous responding significantly alleviating concerns. Some research demonstrates that social desirability is most problematic with students early in their college experience (Bowman & Hill, 2011). Thus, for those looking for the most conservative approach to addressing social desirability when using the SRLS, analytic approaches such as the one employed in Dugan and Komives (2010) using a sample of only college seniors may be most appropriate.…”

“…The halo effect is a well-documented phenomenon in self-report research in which participants' judgments on a particular item are influenced by more general perceptions or the perception of a shared relationship between concepts (Cooper, 1981;Pike, 1999;Bowman & Hill, 2011;Porter, 2011). Several scholars argue that the halo effect potentially increases based on the level of abstraction of the concepts being measured (Gonyea, 2005;Rosch & Schwartz, 2009;Herzog & Bowman, 2011).…”

Experiencing the Unexpected: Toward a Model of College Diversity Experiences and Attitude Change

“…For instance, students' estimates of their gains during college may contain substantial biases (Bowman & Hill, 2011;Pike, 1993;Seifert & Asel, 2011), and the correlations between students' self-reported gains and longitudinal gains on the same constructs are either low or virtually zero (e.g., Bowman, 2010aBowman, , 2011bGosen & Washbush, 1999). Moreover, students are also largely inaccurate when reporting the degree to which a particular experience is associated with their learning and development (Bowman & Seifert, 2011).…”