What response rates are needed to make reliable inferences from student evaluations of teaching?

Zumrawi, Abdel Azim; Bates, Simon; Schroeder, Marianne

doi:10.1080/13803611.2014.997915

Cited by 15 publications

(9 citation statements)

References 14 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Normality of data was assessed via LISREL (Jöreskog & Sörbom 2004) and results showed that data were non-normal with highly skewed distributions, which was an expected situation as average of SET scores tend to be above midpoint (Chandreskar et al, 2013;Zumrawi, 2014). Thus, assessment of fit of baseline model was made based on scaled Chi-square statistic developed by Satorra and Bentler (S-Bχ2) (Satorra & Bentler 1988;Satorra & Bentler 2001), an index typically applied under non-normality and with maximum likelihood estimation for the invariance checks.…”

Section: Analysesmentioning

confidence: 99%

Measurement Invariance of Student Evaluation of Teaching across Groups defined by Course-Related Variables

Kalender¹

2015

IOJES

View full text Add to dashboard Cite

In the present study, comparability of scores from student evaluation of teaching forms was investigated. This is an important issue because scores given by students are used in decision making in higher education institutions. Three course-related variables (grade level, course type, and course credit) were used to define student subgroups. Then, multi-group confirmatory factor analysis was used to assess invariance of factorial structure, factor loadings and factor means across groups. It was found that although a common factorial structure held across groups, fully invariant factor loadings were observed only across instructors who teach different course types. For other groups, only partial invariance of factor loadings was obtained. Analyses also revealed that none of the subgroups had invariant factor means, indicating a possible bias. Results indicate that comparison of instructors based on student ratings may not be valid as it is mostly assumed.

show abstract

Section: Analysesmentioning

confidence: 99%

Measurement Invariance of Student Evaluation of Teaching across Groups defined by Course-Related Variables

Kalender¹

2015

IOJES

View full text Add to dashboard Cite

show abstract

“…While SETs were traditionally administered in a face-to-face format at the end of a teaching period, they are increasingly moving online into both formal and informal modes (Alderman and Melanie, 2012;Otto, Sanford Jr, and Ross 2008). This exacerbates a number of existing concerns about response rates (Zumrawi, Bates, and Schroeder, 2014), various forms of bias (Marsh, 2007;Spooren, Brockx, and Mortelmans, 2013), demographic effects (Macfadyen et al 2015), and potentially negative correlations with desired learning outcomes (Braga, Paccagnella, Pellizzari, 2014). Overall, the field is contested, and practices are often further confused by the way in which this vast and often contradictory literature may be distilled to form, modify, or confirm an academic's existing biases about the validity or invalidity of SETs in their own institutional setting.…”

mentioning

confidence: 99%

Beyond Average: Contemporary statistical techniques for analysing student evaluations of teaching

Kitto

Williams

Alderman

2018

Assessment & Evaluation in Higher Education

View full text Add to dashboard Cite

Student Evaluations of Teaching (SETs) have been used to evaluate Higher Education teaching performance for decades. Reporting SET results often involves the extraction of an average for some set of course metrics, which facilitates the comparison of teaching teams across different organisational units. Here, we draw attention to ongoing problems with the naive application of this approach. Firstly, a specific average value may arise from data that demonstrates very different patterns of student satisfaction. Furthermore, the use of distance measures (e.g. an average) for ordinal data can be contested, and finally, issues of multiplicity increasingly plague approaches using hypothesis testing. It is time to advance the methodology of the field. We demonstrate how multinomial distributions and hierarchical Bayesian methods can be used to contextualise the SET scores of a course to different organisational units and student cohorts, and then show how this approach can be used to extract sensible information about how a distribution is changing in time. We present a report designed to facilitate sense-making for the more complex statistical methodology that we propose, and demonstrate how it can be used to ensure that this more complex methodology is still appropriately used in decision making.

show abstract

“…Across years, respondents included 41 first year (21%), 44 second year (23%), 53 third year (27%,) and 56 fourth year (29%) students. Aligned with the Faculty's benchmark response rates for student evaluations of teaching and courses (18% with variations between years commonplace) 15 and student data collected annually by the Faculty, the total number and demographic breakdown of respondents was representative of the student body.…”

Section: Methodsmentioning

confidence: 99%

Factor Structure Analysis of Pharmacy Students’ Performance on the Health Education Learning Environment Survey

Rusticus

Albon

2022

American Journal of Pharmaceutical Education

View full text Add to dashboard Cite

While high-quality learning environments are increasingly recognized as vital for health professions education programs and student success, there are no tools that have been validated for use within the pharmacy education context. This study seeks to assess whether the six-factor structure of the Health Education Learning Environment Survey (HELES) will replicate in a sample of pharmacy students. Methods. The study was conducted in a Doctor of Pharmacy program offered at a Western Canadian university. A sample of 288 pharmacy students, across two years of data collection, completed the 35-item HELES as an anonymous and online survey.Results. While the six-factor model of the HELES, as a whole, did not replicate through confirmatory factor analysis, a follow-up examination of unidimensionality for the individual subscales of the HELES showed that five of the six subscales met this requirement. One subscale, the work-life balance subscale, was better represented by the dimensions of time management and emotional well-being. Conclusion. These results provide preliminary support for the use of the HELES among pharmacy students, with additional research being needed to explore the work-life balance subscale and its appropriateness for this student group. In Canada, the HELES has the potential to fill an existing local and national gap in available program evaluation tools needed for gathering evidence on pharmacy program quality, strengths, and weaknesses, and to inform continuous quality improvement efforts and accreditation standards.

show abstract

What response rates are needed to make reliable inferences from student evaluations of teaching?

Cited by 15 publications

References 14 publications

Measurement Invariance of Student Evaluation of Teaching across Groups defined by Course-Related Variables

Measurement Invariance of Student Evaluation of Teaching across Groups defined by Course-Related Variables

Beyond Average: Contemporary statistical techniques for analysing student evaluations of teaching

Factor Structure Analysis of Pharmacy Students’ Performance on the Health Education Learning Environment Survey

Contact Info

Product

Resources

About