Measuring Conversational Uptake: A Case Study on Student-Teacher Interactions

Abstract

In conversation, uptake happens when a speaker builds on the contribution of their interlocutor by, for example, acknowledging, repeating or reformulating what they have said. In education, teachers' uptake of student contributions has been linked to higher student achievement. Yet measuring and improving teachers' uptake at scale is challenging, as existing methods require expensive annotation by experts. We propose a framework for computationally measuring uptake, by (1) releasing a dataset of student-teacher exchanges extracted from US math classroom transcripts annotated for uptake by experts; (2) formalizing uptake as pointwise Jensen-Shannon Divergence (PJSD), estimated via next utterance classification; (3) conducting a linguisticallymotivated comparison of different unsupervised measures and (4) correlating these measures with educational outcomes. We find that although repetition captures a significant part of uptake, PJSD outperforms repetitionbased baselines, as it is capable of identifying a wider range of uptake phenomena like question answering and reformulation. We apply our uptake measure to three different educational datasets with outcome indicators. Unlike baseline measures, PJSD correlates significantly with instruction quality in all three, providing evidence for its generalizability and for its potential to serve as an automated professional development tool for teachers.

Proceedings of the 17th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2022)

“…We obtain an average interrater leave-out Spearman correlation of ρ = .644 for UNFILTERED (Fleiss κ = .415 4 ), and ρ = .318 for FILTERED (Fleiss κ = .318). Our interrater agreement values are considered high comparable to those obtained by Demszky et al (2021a) for uptake, and those obtained in widely-used classroom observation protocols such as MQI and the Classroom Assessment Scoring System (CLASS) (Pianta et al, 2008). The lower agreement value for FILTERED indicates that distinguishing funneling vs focusing questions is more subjective than evaluating if a teacher utterance meets the criteria for a follow-up question.…”

“…Recent advances in natural language processing has led to a larger presence of work applying neural methods with varying levels of success in detecting classroom discourse variables, such as semantic content, instructional talk, and elaborated evaluation (Jensen et al, 2021;. For unsupervised approaches, Demszky et al (2021a), which is also most similar to our work in terms of approach and dataset, propose an unsupervised measure of teachers' uptake of students' contributions, and we use their sample in our annotation for funneling and focusing. Other computational work on questions in classroom discourse has focused on detecting questions in live classroom audio (Donnelly et al, 2017;Blanchard et al, 2016) and measuring the authenticity of questions in classroom discourse (Cook, 2018;Kelly et al, 2018).…”

“…We recruit former and current math teachers and educators trained in classroom observation to annotate 2,348 examples of teacherstudent exchanges. We use the same sample of exchanges as Demszky et al (2021a) -they are sampled from transcripts of 45-60 minute long 4th and 5th grade elementary math classroom observations collected by the National Center for Teacher Effectiveness (NCTE) between 2010-2013 (Kane et al, 2015). 2 The transcripts represent data from 317 teachers across 4 school districts in New England that serve largely low-income, historically marginalized students.…”

“…Our annotation framework for funneling vs focusing is designed by experts in math quality instruction, including our collaborators, math teachers and raters for the Mathematical Quality Instruction (MQI) coding instrument, used to assess math instruction (Teaching Project, 2011). We prepare a dataset of utterance pairs (S, T ) for annotation, where S is a student utterance and T is a subsequent teacher utterance following the approach of Demszky et al (2021a). In the annotation interface, raters can see the utterance pair (S, T ), the lesson topic, which is manually labeled as part of the original dataset, and two utterances immediately preceding (S, T ) for context.…”

On the Opportunities and Risks of Foundation Models

“…With the advent of the digital age and the rapid growth in digital learning, computational approaches to education have shown promise in increasing the effectiveness of learners and teachers. Several core directions have emerged as potentially impactful applications of AI for education [Woolf et al 2013], such as systems that can provide meaningful feedback to students [Malik et al 2021], help teachers improve [Jensen et al 2020;Demszky et al 2021;Suresh et al 2021], or even create personalised and adaptive learning experiences that tailor the learning process to individual students' needs and dispositions ].…”