National student survey metrics: where is the room for improvement?

Abstract

The purpose of this study is to use machine learning and exploratory data analysis to interrogate patterns of metrics from a national-level student survey. Analysis of over 1.8 million returns detected long-term stability of the predictors of student satisfaction, with survey items relating to course management and teaching being consistently most influential. All metric outputs increased over the survey period; however, the rates of increase of several dimensions including Overall Satisfaction decreased markedly in the most recent years to a point of levelling off. There was also a growing similarity in an institution of outcomes at a national level. This study contributes new insights into the influential survey instrument, through rigorous determination of the most influential survey items, descriptions of the changes in variability between institutions, and exploration of the importance of patterns of outliers at the extremes of the metric outputs. We also identify a rapidly growing spike in total satisfaction at a broad course level and highlight how this is inconsistent with a customer satisfaction model. We conclude by considering the challenges of the use of national-level student surveys for the management of student satisfaction in higher education.

An Evaluation of an Intervention Programme in Teacher Training for Geography and History: A Reliability and Validity Analysis

“…In direct relation to this aspect, the way in which the topics are approached in class stands out. These results are consistent with studies by Burgess, Senior and Moores [49] and Langan and Harris [50] who found organization, classroom management and teaching quality impacts on and is related to student satisfaction.…”

Subscales in the National Student Survey (NSS): Some considerations on their structure

“…Moreover, much of this work was undertaken on early NSS data. Recent research suggests that there has been a general rise in NSS results over the years leading to a ceiling effect (Burgess et al, 2018;Langan & Harris, 2019). This general rise in satisfaction over the years and ceiling effect may make it more difficult to differentiate between the different factors in the 2019 data that we used for this analysis.…”

“…Third, since the NSS was introduced, there has been a substantial rise in overall student satisfaction across the board (Burgess, Senior, & Moores, 2018;Langan & Harris, 2019), suggesting that this survey has been useful in raising the standards of teaching in higher education. Indeed, many universities and courses use the data from the NSS to improve their student experience.…”

“…Alternatively, individual-items could be used rather than the aggregated data. For example, recent research has demonstrated the effectiveness of using individual-item approaches to identify strategies for improving overall satisfaction (Langan & Harris, 2019). As such, individual-item based approaches can be used effectively to enhance the student experience following feedback from the NSS.…”

A Critical Review of Learning Gains Methods and Approaches

“…However, questions remain whether such measures are adequate representations of actual excellence of teaching, students' learning, skills and/or personal development. Indeed, a wide range of academics have challenged the validity and suitability of those surveys in assessing quality of degree courses at different universities (e.g., Langan & Harris, 2019;Richardson, 2013;Rienties & Toetenel, 2016). For example, in a recent review of 1.8 million NSS returns by Langan and Harris (2019) showed an increasing similarity between institutions in terms of overall satisfaction, leading to questions about validity and usefulness to distinguish good from excellent teaching.…”