Hello GPT! Goodbye home examination? An exploratory study of AI chatbots impact on university teachers’ assessment practices

AI Chatbots in Higher Education. A state-of-the-art review of an emerging research area

“…Although diverse, several of the studies included in this review are exploratory and experimental (Farazouli et al, 2023;Hallal et al, 2023;Jafari & Keykha, 2023;Khosravi et al, 2023), using expert validation techniques to examine the quality of chatbot responses to authentic examination questions that could be asked to students. We also observe that the majority of studies are either observational nature or based on small data sets, or is self-reported and as such cannot be yet used to identify robust ndings or generalizations for university teachers and their practices.…”

“…Broadly, research output or evidence is often ranked from weakest to strongest and presented in the following order: single case studies, cohort studies on groups of people, randomised controlled trials using intervention and control groups to measure the effect of an intervention, evidence syntheses where different types of evidence are combined, and at the top of the hierarchy, systematic reviews where authors have systematically searched for, evaluated, and summarized a comprehensive amount of data to conclude the best available evidence (Farazouli et al, 2023). Hierarchies have been argued to help guide the evaluation of research that predominantly targets effectiveness with an emphasis on experimental and quantitative design, while examining non-causal relationships or engaging in exploratory work requires other methodologies and approaches (Burns & Schuller, 2007).…”

“…We identi ed discourses of utopia-just-around-the-corner in several studies for example Dakakni and Safa (2023) and Rodway and Schepman (2023) highlighting the signi cant advantages of AI in education that support tailored learning experiences, boost and improve student learning and facilitate the identi cation of students' strengths and weaknesses, and subsequently adapting lessons to cater to their individual learning needs. Additionally, focusing on how AI chatbots and their usability were portrayed we identi ed this discourse represented in studies such as ( In Farazouli et al, (2023) for example, chatbots are potentially an existential threat to higher education teachers' assessment practices which we identify as the dystopia-is-now discourse. In Al-Zahrani (2023)…”

“…In Farazouli et al, (2023) for example the ndings suggest teachers lose a sense of control which we understand as an expression of the discourse of altering authority. Here agency is not necessarily only altered but there are concerns that it may be lost.…”

“…in the corpus of articles we included in the review, as well as studies (4) using mixed methods included either in-depth or semi-structured interviews, as part of their data collection (i.e Dakakni & Safa, 2023)(Farazouli et al, 2023;Hallal et al, 2023;Khosravi et al, 2023;Li et al, 2023). Expert validation, where AI generated responses are tested on experts for validity is a form of evidence about how well chatbots perform in higher education settings, and how well they could challenge core practices in education as was evidenced inFarazouli et al, (2023), but does not target students' learning.…”