Informing Expert Feature Engineering through Automated Approaches: Implications for Coding Qualitative Classroom Video Data

Hur, Paul; Machaka, Nessrine; Krist, Christina; Bosch, Nigel

doi:10.1145/3576050.3576090

Cited by 2 publications

(1 citation statement)

References 25 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…On the contrary, the challenge dimension detection models built from TF-IDF and engineered features performed equally at the average accuracy = 0.82, F1 weighted, TF-IDF = 0.80 and F1 weighted, multifeatures = 0.79. This suggests that engineered features may not notably enhance model performance which is consistent with previous research that feature engineering may not always result in eective models, particularly with complex target constructs [24].…”

Section: The Supervised Machine Learning Model Approachsupporting

confidence: 86%

Predicting challenge moments from students' discourse: A comparison of GPT-4 to two traditional natural language processing approaches

Suraworachet,

Seon,

Cukurova

2024

Proceedings of the 14th Learning Analytics and Knowledge Conference

View full text Add to dashboard Cite

Eective collaboration requires groups to strategically regulate themselves to overcome challenges. Research has shown that groups may fail to regulate due to dierences in members' perceptions of challenges which may benet from external support. In this study, we investigated the potential of leveraging three distinct natural language processing models: an expert knowledge rule-based model, a supervised machine learning (ML) model and a Large Language model (LLM), in challenge detection and challenge dimension identication (cognitive, metacognitive, emotional and technical/other challenges) from student discourse, was investigated. The results show that the supervised ML and the LLM approaches performed considerably well in both tasks, in contrast to the rule-based approach, whose ecacy heavily relies on the engineered features by experts. The paper provides an extensive discussion of the three approaches' performance for automated detection and support of students' challenge moments in collaborative learning activities. It argues that, although LLMs provide many advantages, they are unlikely to be the panacea to issues of the detection and feedback provision of socially shared regulation of learning due to their lack of reliability, as well as issues of validity evaluation, privacy and confabulation. We conclude the paper with a discussion on additional considerations, including model transparency to explore feasible and meaningful analytical feedback for students and educators using LLMs.

show abstract

Section: The Supervised Machine Learning Model Approachsupporting

confidence: 86%