MORE – a multimodal observation and analysis system for social interaction research

Abstract

The MORE system is designed for observation and machine-aided analysis of social interaction in real life situations, such as classroom teaching scenarios and business meetings. The system utilizes a multichannel approach to collect data whereby multiple Sami Huttunen Multimed Tools Appl streams of data in a number of different modalities are obtained from each situation. Typically the system collects a 360-degree video and audio feed from multiple microphones set up in the space. The system includes an advanced server backend component that is capable of performing video processing, feature extraction and archiving operations on behalf of the user. The feature extraction services form a key part of the system and rely on advanced signal analysis techniques, such as speech processing, motion activity detection and facial expression recognition in order to speed up the analysis of large data sets. The provided web interface weaves the multiple streams of information together, utilizes the extracted features as metadata on the audio and video data and lets the user dive into analyzing the recorded events. The objective of the system is to facilitate easy navigation of multimodal data and enable the analysis of the recorded situations for the purposes of, for example, behavioral studies, teacher training and business development. A further unique feature of the system is its low setup overhead and high portability as the lightest MORE setup only requires a laptop computer and the selected set of sensors on site.

The evidence of impact and ethical considerations of Multimodal Learning Analytics: A Systematic Literature Review

“…In Martinez-Maldonado, Echeverria, et al (2020), based on 45 minutes of interviews with eight educators and 30 minutes of focus groups with students of all teams, they found that teachers and students appreciated the layers of storytelling. Similarly, MMLA systems were anecdotally argued to have a positive impact on improving teacher reflection (Howard et al, 2019); to increase student interactions in distance learning settings (Yueh et al, 2014); to better guide students' reading behaviours (Huang et al, 2014); to help teachers better observe students conversations in problem-solving activities (Keskinarkaus et al, 2016); to provide real-time feedback in collocated collaboration (Praharaj et al, 2018); to assess the use of mobile mixed reality in health education (Birt et al, 2019); to increase the effective use of LA systems ( Martinez-Maldonado, Elliott, et al, 2020); to develop collaboration and problem-solving skills (Martinez-Maldonado et al, 2019); and to improve task performance (Abrahamson et al, 2016).…”

“…However, the authors did not state any particular privacy concerns related to the use of video data in their research, for instance, the fact that students are more likely to be identified in a video compared to an audio recording. Keskinarkaus et al (2016) suggested processing video and audio data and anonymizing the identity of the students in the future to allow the sharing and using of recorded clips in the discussion while preventing privacy violations. However, they did not discuss the details of the potential implications of these decisions.…”

Exploring groups’ affective states during collaborative learning: what triggers activating affect on a group level?

“…Phases 2-4 were recorded with the MORE observation system, which is designed to observe socialinteraction in real-life contexts (e.g., classroom learning situations). It enables a multichannel approach to collect data simultaneously with three 360° cameras and multiple microphones (for technical information, see Keskinarkaus et al, 2016). In this case, three groups worked simultaneously in the LeaForum space while their collaboration was recorded with one 360° camera for each group and an individual microphone for each student.…”

Emotion recognition and school violence detection from children speech

“…Data was collected from pupils using different procedures including pre-and post-interviews and video analysis. The activities of the groups were video recorded by MORE recording system (Multimodal Recording System) for the later analysis [15]. The emotional speeches in task three are also recorded by MORE recording system.…”