“It’s Weird That it Knows What I Want”: Usability and Interactions with Copilot for Novice Programmers

Abstract

Recent developments in deep learning have resulted in code-generation models that produce source code from natural language and code-based prompts with high accuracy. This is likely to have profound effects in the classroom, where novices learning to code can now use free tools to automatically suggest solutions to programming exercises and assignments. However, little is currently known about how novices interact with these tools in practice. We present the first study that observes students at the introductory level using one such code auto-generating tool, Github Copilot, on a typical introductory programming (CS1) assignment. Through observations and interviews we explore student perceptions of the benefits and pitfalls of this technology for learning, present new observed interaction patterns, and discuss cognitive and metacognitive difficulties faced by students. We consider design implications of these findings, specifically in terms of how tools like Copilot can better support and scaffold the novice programming experience.

Prompt Problems: A New Programming Exercise for the Generative AI Era

“…Over-reliance on AI generated outputs is a commonly cited concern within the education community, and several students commented on this aspect, including: "it is critical for students to learn the ability to write code independently rather than relying only on AI-generated answers" and "I feel like it is too tempting of a tool to use through the labs and not learn and develop these skills yourself". These concerns align with previous work that has looked into students' opinions on AI code generation [25].…”

“…Other studies on the capabilities of LLMs have revealed impressive proficiency in dealing with object-oriented programming tasks [5], Parsons problems [26], and compiler error messages [17]. Many of these explorations also revealed that LLMs are not infallible and can produce solutions that do not align with best programming practice [5], struggle with longer and higher-level specifications [12], and cause students to become confused reading code that they did not write themselves [14,25]. Babe et al showed that LLMs can mislead students, causing them to believe that their own prompts are more (or less) effective than they are in reality [2].…”

Enhancing Programming Error Messages in Real Time with Generative AI

“…Vaithilingam et al looked at very early usage of students using LLMs to code and found that students preferred using it, despite it not decreasing their overall task completion times [29]. The fact that it didn't make novices any faster at solving programming problems could be because students struggle to understand code that has been autogenerated for them, as found by Prather et al [23]. Kazemitabaar et al found that students will utilize it to either explore new ways of doing a task or to attempt to accelerate their task completion time [10].…”

“…Researchers have been quick to measure the ability of these LLMs with regard to typical computer science programs [7,8], code explanations [12], exams [17], and even Parsons Problems (mixed-up code problems) [24]. This has led to instructors having many concerns, such as students using autogenerated code that they don't understand [23] or using these tools to do their work for them [1,11]. Still other concerns, such as over-reliance, biases inherent in the models, and trustworthiness, remain active points of discussion in current research [31].…”

CodeAid: Evaluating a Classroom Deployment of an LLM-based Programming Assistant that Balances Student and Educator Needs

“…Future research is also needed to evaluate the impact of proactive problem identification and assistance, focusing on the design of minimally distracting assistants. Indeed, recent work by Prather et al revealed that students did not like being shown suggestions when they felt they did not need the help [51].…”