Exploring Kid Space in the wild: a preliminary study of multimodal and immersive collaborative play-based learning experiences

Abstract

Parents recognize the potential benefits of technology for their young children but are wary of too much screen time and its potential deficits in terms of social engagement and physical activity. To address these concerns, related literature suggests technology usages with a blend of digital and physical learning experiences. Towards this end, we developed Kid Space, incorporating immersive computing experiences designed to engage children more actively in physical movement and social collaboration during play-based learning. The technology features an animated peer learner, Oscar, who aims to understand and respond to children’s actions and utterances using extensive multimodal sensing and sensemaking technologies. To investigate student engagement during Kid Space learning experiences, an exploratory case study was designed using a formative research method with eight first-grade students. Multimodal data (audio and video) along with observational, interview, and questionnaire data were collected and analyzed. The results show that the students demonstrated high levels of engagement, less attention focused on the screen (projected wall), and more physical activity. In addition to these promising results, the study also enabled us to understand actionable insights to improve Kid Space for future deployments (e.g., the need for real-time personalization). We plan to incorporate the lessons learned from this preliminary study and deploy Kid Space with real-time personalization features for longer periods with more students.

What is the impact of a multi‐modal pedagogical conversational AI system on parents' concerns about technology use by young children?

“…However, for this first study, we utilised a Wizard-Of-Oz setup to collect authentic data for fine-tuning the AI algorithms and game logic prior to future real-world pilot deployments. For more details about the wizard interface, refer to Aslan, Agrawal, et al (2022).…”

“…We followed the labelling protocol by Aslan et al (2017) for the expert coders to annotate the data for different pillars of learner engagement: (1) task-engagement (whether the learner is on-task or not), (2) focus of attention (where the learner is looking), (3) physical activity (whether the learner is physically active or stationary) and ( 4) social interaction (whether the learner is verbally or non-verbally interacting with the virtual agent, or not). For more details about operational definitions of these labels, see Aslan, Agrawal, et al (2022).…”

NLU for Game-based Learning in Real: Initial Evaluations

“…Planting game and Watering game POC datasets have 1927 and 2115 utterances, respectively. The deployment game datasets were later collected from 12 kids (and two adults), where the Kid Space setup was deployed in a classroom at school (Aslan et al, 2022). For both in-the-lab and in-the-wild datasets, the spoken user utterances are transcribed manually at first.…”

Data Augmentation with Paraphrase Generation and Entity Extraction for Multimodal Dialogue System

“…The system needs to be physically grounded to allow children to bring meaningful objects into the play experience, such as physical toys and manipulatives as learning materials. Thus, Kid Space AI system would combine various sensing technologies that should interact with children, track each child, and monitor their progress (Anderson et al, 2018;Aslan et al, 2022). The use-cases contain specific flows of interactive games facilitating elementary math learning designed for children 5-to-8 years old.…”

“…We conduct our experiments on the KidSpace NLU datasets having utterances from multimodal math learning experiences (i.e., Planting and Watering activities) designed for 5-to-8 years-old kids (Sahay et al, 2021;Aslan et al, 2022). These are the initial proof-of-concept (POC) datasets to bootstrap the agents to be deployed.…”