Impersonating Chatbots in a Code Review Exercise to Teach Software Engineering Best Practices

Abstract

Over the past decade, the use of chatbots for educational purposes has gained considerable traction. A similar trend has been observed in social coding platforms, where automated agents support software developers with tasks such as performing code reviews. While incorporating code reviews and social coding platforms into software engineering education has been found to be beneficial, challenges such as steep learning curves and privacy considerations are barriers to their adoption. Furthermore, no study has addressed the role chatbots play in supporting code reviews as a pedagogical tool. To help address this gap, we developed an online learning application that simulates the code review features available on social coding platforms and allows instructors to interact with students using chatbot identities. We then embedded this application within a lesson on software engineering best practices and conducted a controlled in-class experiment. This experiment examined the effect that explaining content via chatbot identities had on three aspects: (i) students' perceived usability of the lesson, (ii) their engagement with the code review process, and (iii) their learning gains. While our findings show that it is feasible to simulate the code review process within an online learning platform and achieve good usability, our quantitative analysis did not yield significant differences across treatment conditions for any of the aspects considered. Nevertheless, our qualitative results suggest that students expect explicit feedback when performing this type of exercise and could thus benefit from automated replies provided by an interactive chatbot. We propose to build on our current findings to further explore this line of research in future work.

AI-powered peer review process

“…We found that genAI stimulated student engagement in the code review process, demonstrating a higher average number of identified issues compared to the peer review process and a higher average number of those issues identified by genAI were fixed. This aligns with results reported in previous studies that used automated peer feedback (Farah et al, 2022). Interestingly, inaccurate feedback and misunderstandings of project requirements encouraged students to revisit the inspected code and stimulated further critical discussion.…”

“…By (partially) delegating code review responsibilities to AI-based tools, students can review and reflect on their coding practices without feeling exposed. Several previous studies investigated the use of automated code review in software engineering (Indriasari et al, 2020;Kaufmann et al, 2022;Farah et al, 2022), however, to the best of our knowledge, no previous work has investigated the use of generative AI combined with assessment checklists in educational code review processes.…”

A Blueprint for Integrating Task-Oriented Conversational Agents in Education

“…As chatbots have been particularly useful in computer-assisted language learning (CALL) [2,7,31], it would be particularly pertinent to implement chatbots aimed at supporting tasks in CALL (e.g., providing conversational practice, checking grammar, interfacing dictionaries). These implementations will be validated following experimental designs used in previous exploratory studies [11,12,14] comprising controlled experiments measuring learning gains, engagement, and usability using both standard (e.g., the User Experience Questionnaire [24] and the System Usability Scale [3]) and bespoke (e.g., scores calculated using the results of pre/post-tests) instruments. Third, the need to support instructors in the configuration and deployment of chatbots built with our blueprint was not addressed.…”

Harnessing Rule-Based Chatbots to Support Teaching Python Programming Best Practices

“…However, very few studies have focused on the use of chatbots to support the code review process in formal education settings. We build on a previous Wizard of Oz [6] experiment [10] and a pilot study [8] to explore the impact that rule-based chatbots supporting code review exercises could have on the learning experience. In the following section, we present our guiding research question and the methodology we followed for our evaluation.…”