Measuring Learning Outcomes in Higher Education

Abstract

With the pressing need for accountability in higher education, standardized outcomes assessments have been widely used to evaluate learning and inform policy. However, the critical question on how scores are influenced by students’ motivation has been insufficiently addressed. Using random assignment, we administered a multiple-choice test and an essay across three motivational conditions. Students’ self-report motivation was also collected. Motivation significantly predicted test scores. A substantial performance gap emerged between students in different motivational conditions (effect size as large as .68). Depending on the test format and condition, conclusions about college learning gain (i.e., value added) varied dramatically from substantial gain (d = 0.72) to negative gain (d = −0.23). The findings have significant implications for higher education stakeholders at many levels.

Motivation Filtering on a Multi-Institution Assessment of General College Outcomes

“…That is, there were apparently lower levels of motivation among second-semester sophomores compared to entering freshmen. This finding bolsters concerns that longitudinal effect sizes reported elsewhere (e.g., Arum & Roksa, 2011) may be depressed because of student motivation (Liu, Bridgeman, & Adler, 2012 Whether test results are used to guide the improvement of educational programs or for highstakes accountability purposes, it is imperative that aggregate test scores provide indications of student skills and knowledge that can be interpreted validly. Motivation filtering holds some promise for improving the validity of aggregate test score interpretations, especially when results are used internally by institutions to inform relatively low-stakes decisions related to improving educational programs.…”

Value-added assessment in higher education: a comparison of two methods

“…Second, the current method groups all types of institutions together for value-added analysis without considering any institutional characteristics. However, many factors, such as institution selectivity and type could have a profound impact on student learning, and should be considered in determining student progress (Liu 2008(Liu , 2009aBorden and Young 2008). Finally, the current method uses OLS regression models to analyze student performance on outcomes tests.…”

The Role of Students’ Attitudes and Test-Taking Motivation on the Validity of College Institutional Accountability Tests: A Path Analytic Model

“…Although not always explicitly stated, many strategies for boosting students' test-taking effort focus on increasing perceived test importance or attitudes that will, in turn, influence perceived test importance (e.g., Wise, 2009;Wise & Cotten, 2009). That is, some studies have suggested manipulating the stakes associated with the test via test instructions to improve test-taking motivation and performance (e.g., Liu, Bridgeman, & Adler, 2012). However, for many low-stakes tests in the higher education context, K-12 context (e.g., National Assessment of Educational Progress), and international context (e.g., PISA, TIMSS), instructions noting personal consequences would not be truthful and thus not ethical.…”