Citation: KING, M. ...et al., 2017. Utilizing a realist evaluative research approach to investigate complex technology implementations: an e-learning lecture capture exemplar.Journal of Systems and Information Technology, 19(1/2), pp.22-41.Additional Information:• This paper was accepted for publication in the journal Journal of Sys- AbstractPurpose -This study aims to investigate the effectiveness of a theory-driven realist evaluative research approach to better understand complex technology implementations in organisations.Approach -An institution wide e-learning implementation of Lecture Capture (LC), within a UK University, was chosen and a realist evaluation framework was used, tailored for educational technology. The research was conducted over 4, increasingly focused, evaluation cycles combining engagement analytics, user interviews and theory to refine what works (or does not work), for whom, in which contexts and why.Findings -Despite explicit demand and corresponding investment, overall student engagement is lower than expected. Increased student use appears linked to particular staff attitudes and behaviours and not to specific disciplines or course content. The main benefits of LC are; providing reassurance to the majority, aiding revision and understanding for the many, and enabling catch-up for the few. Recommendations for future research are based on some unexpected outcomes uncovered, including; evolving detrimental student behaviours, policy development based on technological determinism and future learner-centred system development for next-generation LC technologies.Practical implications -The realist approach taken, and evaluation framework used, can be adopted (and adapted) for future evaluative research. Domain specific reference models, categorizing people and technology, supported analysis across multiple contexts.Originality/value -This study responds to a call for more theory-based research in the field of educational technology. We demonstrate that a theory-driven approach provides real and practical recommendations for institutions and allows for greater insight into the political, economic and social complexity of technology implementation.

The increased presence of diagram-type student work in higher education has recently attracted researchers to look into the automation of diagram marking. This paper intro duces web-based diagram drawing and marking tools for a new (semi-automatic) assessment approach. The approach reduces the number of diagram components marked by the human marker and provides individualised and detailed feedback to students. The tools have been used in tutorials of a first year database module in the Computer Science department at Loughborough University are described together with findings from the usage of the tools.

In recent years, many students in higher education have begun to learn programming languages. In doing so they will complete a variety of programming tasks of varying degrees of complexity. The students need to get consistent and personalized feedback to develop their programming skills. Human markers can provide personalized feedback using traditional manual approaches to assessment, but they may provide inconsistent feedback (especially for long programming solutions) since marking the programming solutions of multiple students can represent a significant workload for them. While full‐automated assessment systems are the best to provide consistent feedback, they may not provide sufficiently personalized feedback for novice programmers. This study develops a novel semi‐automated assessment approach in order to improve efficiency of human marker in the marking process and increase consistency of feedback (for both short and long programming solutions). It advocates the reuse of human marker's comments for similar code snippets, defined as segmented marking in this study. New full and partial marking models are developed based on segmented marking and they are tested by expert markers. The findings show that the two models are similar in efficiency, but that a partial marking approach potentially offers an improved efficiency for longer programming solutions. Such a finding has significant potential to reduce time spent on marking throughout the sector, which would have significant impact on both resourcing and timeliness of feedback.

No abstract

No abstract