What if Social Robots Look for Productive Engagement?

Abstract

In educational HRI, it is generally believed that a robots behavior has a direct effect on the engagement of a user with the robot, the task at hand and also their partner in case of a collaborative activity. Increasing this engagement is then held responsible for increased learning and productivity. The state of the art usually investigates the relationship between the behaviors of the robot and the engagement state of the user while assuming a linear relationship between engagement and the end goal: learning. However, is it correct to assume that to maximise learning, one needs to maximise engagement? Furthermore, conventional supervised models of engagement require human annotators to get labels. This is not only laborious but also introduces further subjectivity in an already subjective construct of engagement. Can we have machine-learning models for engagement detection where annotations do not rely on human annotators? Looking deeper at the behavioral patterns and the learning outcomes and a performance metric in a multi-modal data set collected in an educational human–human–robot setup with 68 students, we observe a hidden link that we term as Productive Engagement. We theorize a robot incorporating this knowledge will (1) distinguish teams based on engagement that is conducive of learning; and (2) adopt behaviors that eventually lead the users to increased learning by means of being productively engaged. Furthermore, this seminal link paves way for machine-learning models in educational HRI with automatic labelling based on the data.

Many are the ways to learn identifying multi-modal behavioral profiles of collaborative learning in constructivist activities

“…Further, we refer to multi-modality as the application and interplay of multiple semiotic modes in order to help understand a specific process, in this case, learning. In previous work (Nasir, Bruno, et al, 2021a), we found that in an open-ended collaborative learning activity, multimodal behaviors (extracted from video, audio, and log data) better distinguish those who learn from those who do not compared to when only a single modality was used. Further, we argue that it is not straightforward to classify a certain behavior as absolutely good or bad for learning.…”

“…This dataset, with multi-modal behaviors as well as performance metrics and learning gains, has been made publicly available (Nasir, Bruno, et al, 2021a). It must be noted that, in our dataset, all behaviors are treated as cross-sectional averages and frequencies (i.e., over the entire duration of the task), and thus it is not time series data.…”

“…To this end, we developed an analysis approach consisting of two parts: a quantitative approach and a qualitative approach. The quantitative approach relies on the outcomes of our previously published learning analytics technique (Nasir et al, 2021a, b;Nasir, Norman, Bruno, & Dillenbourg, 2020c) that helps identify groups of learners who have learning gains and those who don't. Using this approach, we are able to build the multimodal behavioral profiles for each group of learners.…”

Studying Alignment in a Collaborative Learning Activity via Automatic Methods: The Link Between What We Say and Do

“…At a higher level, lack of understanding/learning etc. could be reflected by other phenomena (such as other multimodal features (Nasir et al, 2020a(Nasir et al, , 2021), other expressions ("I don't understand") etc. Since there are many ways to learn, and hence different behaviours that could result in learning, it is unsurprising that such patterns are difficult to capture.…”

Behavioral Data Analysis of Robot-Assisted Autism Spectrum Disorder (ASD) Interventions Based on Lattice Computing Techniques

“…In a recent study [23], a post-intervention analysis on multi-modal data collected during learning activities involving a humanoid robot and two children was presented. The authors extracted engagement-related features from video, audio and log data in order to establish correspondences between clustered pairs of students and their corresponding behavior patterns and learning labels.…”