Example-based feedback provision using structured solution spaces

Abstract

Intelligent tutoring systems (ITSs) typically rely on a formalised model of the underlying domain knowledge in order to provide feedback to learners adaptively to their needs. This approach implies two general drawbacks: the formalisation of a domain-specific model usually requires a huge effort, and in some domains it is not possible at all. In this paper, we propose feedback provision strategies in absence of a formalised domain model, motivated by example-based learning approaches. We demonstrate the feasibility and effectiveness of these strategies in several studies with experts and students. We discuss how, in a set of solutions, appropriate examples can be automatically identified and assigned to given student solutions via machine learning techniques in conjunction with an underlying dissimilarity metric. The plausibility of such an automatic selection is evaluated in an expert survey, while possible choices for domain-agnostic dissimilarity measures are tested in the context of real solution sets of Java programs. The quantitative evidence suggests that the proposed feedback strategies and

Adaptive structure metrics for automated feedback provision in intelligent tutoring systems

“…However, researchers have reported 100 -1,000 hours of authoring time for one hour of instructions in ITSs [3]; in addition, ITSs usually require an underlying domain theory such that their applicability is limited in areas where problems and their solution strategies are not easy to formalize [4,5]. In such domains, data-driven approaches are possible, providing feedback based on a set of existing examples for (correct) solutions of the underlying task [5,6]: If the students requires a hint on how to change her attempt to get closer to a correct solution, it can be compared to a similar example from the set, and the dissimilarities between her attempt and the example can be contrasted or highlighted in order to help the student to improve her own solution [7,8,9].…”

Learning Feedback in Intelligent Tutoring Systems

“…1). The results show a positive impact of the instruction that was positively rated by the learners [10], and the study also confirmed that manual feedback generation indeed means a lot of work, and that the ITS feedback methods have been assessed as potentially helpful in most cases [7]. Finally, we conducted a field study in an introductory programming course for Java programming, comparing the appropriateness of different exemplar selection strategies (user or sample solutions, full or partial solutions) for feedback provision.…”

“…A subset of the data set was manually coded in terms of quality for the purpose of later comparison with automated assessments. Also, we defined the term "solution space" in more detail, including both a technical dimension on how to analyze such spaces of learner (and sample) solutions using machine learning techniques, and a pedagogical dimension on how to give useful feedback to a learner's solution within such a space [10]. In addition, we identified several evaluation methods (e.g., pre-/post-testing) and criteria (e.g., learner's action after and before next instruction) that are suitable for measuring the impact of FIT ITS methods on human learning.…”

“…Therein, prototypes are represented implicitly by linear combinations in the underlying pseudo-Euclidean space. We have shown that these relational clustering and classification techniques can structure solution spaces, and yield prototypical solutions for a direct interpretation of the data-driven model [15,10]. Regarding objective 2, we developed a domainagnostic dissimilarity measure, based on a highly parameterized sequence alignment 1 over traversals of subgraphs.…”

“…We then conducted two Wizard-of-Oz studies [10,7] asking experts to simulate the behavior of a FIT ITS. In these studies, we examined two different example-based feedback provision strategies in the domain of Java programming: highlighting and/or contrasting similarities among learner solutions and sample solutions (as illustrated in Fig.…”