Data-driven proficiency profiling

Mostafavi, Behrooz; Barnes, Tiffany

doi:10.1145/2883851.2883935

Cited by 6 publications

(4 citation statements)

References 6 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This work assesses the impact of incorporating hint usage in predicting student behavior. While researchers have extensively explored predictive student modeling tasks such as predicting student performance (Wang and Heffernan 2011;Chaudhry et al 2018), problem selection (Mostafavi et al 2015), and adaptive assistance (Conati et al 2002), less attention has been paid to the role of students' hint usage in such modeling tasks. The work by Beck et al (2004) was one of the first to assess the impact of tutor help on predictive modeling.…”

Section: Hint Usage In Student Modelingmentioning

confidence: 99%

Enhancing a student productivity model for adaptive problem-solving assistance

Maniktala

Chi

Barnes

2022

User Model User-Adap Inter

Self Cite

View full text Add to dashboard Cite

Research on intelligent tutoring systems has been exploring datadriven methods to deliver effective adaptive assistance. While much work has been done to provide adaptive assistance when students seek help, they may not seek help optimally. This had led to the growing interest in proactive adaptive assistance, where the tutor provides unsolicited assistance upon predictions of struggle or unproductivity. Determining when and whether to provide personalized support is a well-known challenge called the assistance dilemma. Addressing this dilemma is particularly challenging in open-ended domains, where there can be several ways to solve problems. Researchers have explored methods to determine when to proactively help students, but few of these methods have taken prior hint usage into account. In this paper, we present a novel data-driven approach to incorporate students' hint usage in predicting their need for help. We explore its impact in an intelligent tutor that deals with the open-ended and well-structured domain of logic proofs. We present a controlled study to investigate the impact of an adaptive hint policy based on predictions of HelpNeed that incorporate students' hint usage. We show empirical evidence to support that such a policy can save students a significant amount of time in training, and lead to improved posttest results, when compared to a control without proactive interventions. We also show that incorporating students' hint usage significantly improves the adaptive hint policy's efficacy in predicting students' HelpNeed, thereby reducing training unproductivity, reducing possible help avoidance, and increasing possible help appropriateness (a higher chance of receiving help when it was likely to be needed). We conclude with suggestions on the domains that can benefit from this approach as well as the requirements for adoption.

show abstract

Section: Hint Usage In Student Modelingmentioning

confidence: 99%

Enhancing a student productivity model for adaptive problem-solving assistance

Maniktala

Chi

Barnes

2022

User Model User-Adap Inter

Self Cite

View full text Add to dashboard Cite

show abstract

“…However, as highlighted by recent reports, such tools may provide little added value on their own. To be effective, they must be well implemented and well integrated into the broader system (Lipsky 2010;Mostafavi, Liu, and Barnes 2015). Ultimately, the success of such tools depends on their adoption by the counselors (Lipsky 2010).…”

Section: Algorithmic Decision-making Long-term Unemployment and Human Detachment And Attachment To Decision-makingmentioning

confidence: 99%

“…For example, a study of Poland's PES system revealed that case counselors are not always transparent about the use of the profiling system with the unemployed (Sztandar-Sztanderska and Zielenska 2018). Several studies analyzed the effectiveness of profiling systems (Lipsky 2010;Mostafavi, Liu, and Barnes 2015;Weber 2011) and how profiling tools are used by counselors in practice (Weber 2011;Sztandar-Sztanderska and Zielenska 2018). However, while the extant scholarly work shows that frontline employees struggle to use the systems that employ AMD with poorly implemented off-the-shelf software interfaces (e.g., Raso 2017), the evidence base is still weak and arguably incomplete.…”

Section: Algorithmic Decision-making Long-term Unemployment and Human Detachment And Attachment To Decision-makingmentioning

confidence: 99%

Algorithmic Long-Term Unemployment Risk Assessment in Use: Counselors’ Perceptions and Use Practices

Zejnilović

Lavado

Martínez

et al. 2020

Global Perspectives

View full text Add to dashboard Cite

The recent surge of interest in algorithmic decision-making among scholars across disciplines is associated with its potential to resolve the challenges common to administrative decision-making in the public sector, such as greater fairness and equal treatment of each individual, among others. However, algorithmic decision-making combined with human judgment may introduce new complexities with unclear consequences. This article offers evidence that contributes to the ongoing discussion about algorithmic decision-making and governance, contextualizing it within a public employment service. In particular, we discuss the use of a decision support system that employs an algorithm to assess individual risk of becoming long-term unemployed and that informs counselors to assign interventions accordingly. We study the human interaction with algorithms in this context using the lenses of human detachment from and attachment to decision-making. Employing a mixed-method research approach, we show the complexity of enacting the potentials of the data-driven decision-making in the context of a public agency.

show abstract

“…Deep Thought (DT, Figure 1) is an intelligent tutor for solving open-ended multi-step propositional logic problems that has data-driven features including next-step hints [8,75], as well as adaptive problem selection [51,53] and ped-Fig. 2: A sample solution of a training problem in Deep Thought agogical policies for worked example presentation induced via reinforcement learning [6,52,72,73].…”

Section: System Designmentioning

confidence: 99%

Avoiding Help Avoidance: Using Interface Design Changes to Promote Unsolicited Hint Usage in an Intelligent Tutor

Maniktala,

Cody,

Barnes

et al. 2020

Preprint

Self Cite

View full text Add to dashboard Cite

Within intelligent tutoring systems, considerable research has investigated hints, including how to generate data-driven hints, what hint content to present, and when to provide hints for optimal learning outcomes. However, less attention has been paid to how hints are presented. In this paper, we propose a new hint delivery mechanism called "Assertions" for providing unsolicited hints in a data-driven intelligent tutor. Assertions are partiallyworked example steps designed to appear within a student workspace, and in the same format as student-derived steps, to show students a possible subgoal leading to the solution. We hypothesized that Assertions can help address the well-known hint avoidance problem. In systems that only provide hints upon request, hint avoidance results in students not receiving hints when they are needed. Our unsolicited Assertions do not seek to improve student helpseeking, but rather seek to ensure students receive the help they need. We contrast Assertions with Messages, text-based, unsolicited hints that appear after student inactivity. Our results show that Assertions significantly increase unsolicited hint usage compared to Messages. Further, they show a significant aptitude-treatment interaction between Assertions and prior proficiency, with Assertions leading students with low prior proficiency to generate shorter (more efficient) posttest solutions faster. We also present a clustering analysis that shows patterns of productive persistence among students with low prior knowledge when the tutor provides unsolicited help in the form of Assertions. Overall, this work provides encouraging evidence that hint presentation can significantly impact how students use them and using Assertions can be an effective way to address help avoidance.

show abstract

Data-driven proficiency profiling

Cited by 6 publications

References 6 publications

Enhancing a student productivity model for adaptive problem-solving assistance

Enhancing a student productivity model for adaptive problem-solving assistance

Algorithmic Long-Term Unemployment Risk Assessment in Use: Counselors’ Perceptions and Use Practices

Avoiding Help Avoidance: Using Interface Design Changes to Promote Unsolicited Hint Usage in an Intelligent Tutor

Contact Info

Product

Resources

About