Why Assessment Will Never Work at Many Business Schools

Abstract

On the long and arduous journey toward effective educational assessment, business schools have progressed in their ability to clearly state measurable learning goals and use direct measures of student learning. However, many schools are wrestling with the last stages of the journey-measuring present learning outcomes, implementing curricular/pedagogical changes, and then measuring postchange outcomes to determine if the implemented changes produced the desired effect. These last steps are particularly troublesome for a reason unrecognized in the assessment literature-inadequate statistical power caused primarily by the use of small student samples. Analyses presented here demonstrate that assessment efforts by smaller schools may never provide the statistical power required to obtain valid results in a reasonable time frame. Consequently, decisions on curricular and pedagogical change are too often based on inaccurate research findings. Rather than waste time and resources toward what essentially is a statistical dead end, an alternate approach is recommended: Schools should examine published pedagogical studies that use direct measures of learning with sufficient statistical power and utilize the findings to improve student learning.

Statistical Significance and Program Effect

“…Colleges and universities have imposed the accountability mechanism—accreditation—on themselves to head off government intervention. 2 Hence outside pressure, while having moved business schools to assess student learning as Bacon and Stewart (2016) note, are also often viewed by administrators and especially faculty as a nuisance—a force to comply with at least symbolically if not substantively.…”

“…Capacity to assess, experiment, and make change is, as Bacon and Stewart (2016) note, often lacking. Most faculty and administrators are not experts in assessment development, psychometrics, experimental design, statistical analysis of behavioral experiments, and using evidence from experiments to improve programs.…”

“…. ., " Bacon and Stewart (2016) recommend that instead of carrying out the expensive process of experimenting themselves, many business schools would get a bigger bang for their buck if they used "published pedagogical studies that use direct measures of learning with sufficient statistical power" (p. 181) to improve instructional programs and student learning. This is because, according to Bacon and Stewart (2016), such studies are doomed to fail the inferential test of significance due to lack of statistical power-a consequence of small enrollments in programs and the unreliability of faculty-made measures of student achievement.…”

Assessing Leader Development: Lessons From a Historical Review of MBA Outcomes

“…Yet programs rarely assess the value added by competency development efforts. Even when assessment is conducted, many programs lack the psychometric rigor and/or sufficient sample size necessary to draw inferences that provide meaningful information on which to base educational innovations (Bacon & Stewart, 2016). Furthermore, outcomes are generally examined on a cohort-by-cohort basis to satisfy accreditation or other compliance demands.…”

Educational Evaluation and Codeterminations of School Production

“…Recent research accounts for the above. Business schools have the strategy of examining and implementing pedagogical studies based on the measurement of learning outcomes with sufficient statistical power Bacon and Stewart (2017). Although the evaluation process is not limited on identifying student knowledge (Santiago et al, 2017), it also serves as a mediator to balance different types of data and knowledge flows (Ellis and Smith, 2017).…”