In the context of increased emphasis on quality assurance of teaching, it is crucial that student evaluation of teaching (SET) methods are both reliable and workable in practice. Especially, online SET tends to raise criticisms with those most reactive to mechanisms of teaching accountability. However, most studies have been conducted with convenience, small and cross-sectional samples. Longitudinal studies are rare, as comparison studies on SET methodological approaches are generally pilot studies followed shortly after by implementation. The investigation presented here significantly contributes to the debate by examining the impact of the online administration method of SET on a very large longitudinal sample at course level. The study explores the impact of the administration method of student evaluations of teaching (paper based in-class versus off-class online collection) on scores with a longitudinal sample of over 63,000 student responses collected over a total period of ten years. Having adjusted for the confounding effect of class size, faculty, year of evaluation, years of teaching experience and student performance, it is observed that the actual effect of the administration method exists, but is insignificant.