The social signal interpretation framework (SSI) for real time signal processing and recognition

Wagner, Johannes; Lingenfelser, Florian; André, Elisabeth

doi:10.21437/interspeech.2011-818

Cited by 19 publications

(3 citation statements)

References 12 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Further, improvements in CIMLA's documentation would help to better communicate CIMLA to stakeholders so that the communication problems that occurred during our evaluation can be resolved. 3 [Munoz et al, 2018] 3 [Tamura et al, 2019] 3 [Ciordas-Hertel et al, 2019] 2 [Huertas Celdrán et al, 2020] 3 [Camacho et al, 2020] 3 [Domínguez et al, 2021] 3 [Serrano Iglesias et al, 2021] 4 [Slupczynski and Klamma, 2021] 2 [Di Mitri et al, 2017] 3 [Muñoz-Cristóbal et al, 2018] 3 [Fiaidhi, 2014] 11 [Domínguez and Chiluiza, 2016] 2 [Ruffaldi et al, 2016] 3 [Harrer, 2013] 2 [Berg et al, 2016] 4 [Wagner et al, 2011] 3 [Segal et al, 2017] 2…”

Section: Discussionmentioning

confidence: 99%

“…We reviewed 18 data infrastructures in MMLA (nine from our last review [Di Mitri et al, 2017, Muñoz-Cristóbal et al, 2018, Fiaidhi, 2014, Domínguez and Chiluiza, 2016, Ruffaldi et al, 2016, Harrer, 2013, Berg et al, 2016, Wagner et al, 2011, Segal et al, 2017 and nine proposed after our review , Munoz et al, 2018, Tamura et al, 2019, Ciordas-Hertel et al, 2019, Huertas Celdrán et al, 2020, Camacho et al, 2020, Domínguez et al, 2021, Serrano Iglesias et al, 2021, Slupczynski and Klamma, 2021). We started our review by finding out how many learning tools were supported in the proposals.…”

Section: Existing Mmla Infrastructuresmentioning

confidence: 99%

“…[REQ1] [F] Fetch datasets from different data repositories to access heterogeneous datasets that store multimodal educational data [Di Mitri, 2020] [REQ2] [F] Unify all the datasets into a single file format [Di Mitri, 2020, Di Mitri et al, 2019a [REQ3] [F] Consider structured and organized CI in the data processing pipeline to support contextaware data processing , Shankar et al, 2019 [REQ4] [F] Process multimodal educational data using a standard data processing pipeline , Di Mitri et al, 2019a, Shankar et al, 2019, Di Mitri et al, 2019b, Hassan et al, 2021 [F] Export output data file in a set of file formats, so that stakeholders can get the data file in their required format [Di Mitri, 2020, Di Mitri et al, 2019a, Shankar et al, 2019 [REQ6] [NF] Support modularity in data processing, so that the complexity of multimodal educational data processing can be broken down [Di Mitri et al, 2019a, Shankar et al, 2019, Hassan et al, 2021, Spikol et al, 2018 [REQ7] [NF] Support modifiability in different components, so that developers do not need to develop them from scratch , Di Mitri et al, 2019a, Shankar et al, 2019 Table 1: A list of functional and non-functional requirements for CIMLA [F -Functional, NF -Non-Functional] To start with the architectural design of CIMLA, we followed a development architectural viewpoint [Woods and Rozanski, 2005] because MMLA developers are the main stakeholders to be supported by CIMLA. Moreover, we followed the Separation-of-Concerns (SoC) architectural design principle to divide the different interests involved in the challenge of developing a context-aware MMLA solution [Hürsch and Lopes, 1995].…”

Section: Tag / Category Requirement Referencesmentioning

confidence: 99%

See 2 more Smart Citations

CIMLA: A Modular and Modifiable Data Preparation, Organization, and Fusion Infrastructure to Partially Support the Development of Context-aware MMLA Solutions

Shankar¹,

Ruiz‐Calleja²,

Prieto³

et al. 2023

jucs

View full text Add to dashboard Cite

Multimodal Learning Analytics (MMLA) solutions aim to provide a more holistic picture of a learning situation by processing multimodal educational data. Considering contextual information of a learning situation is known to help in providing more relevant outputs to educational stakeholders. However, most of the MMLA solutions are still in prototyping phase and dealing with different dimensions of an authentic MMLA situation that involve multiple cross-disciplinary stakeholders like teachers, researchers, and developers. One of the reasons behind still being in prototyping phase of the development lifecycle is related to the challenges that software developers face at different levels in developing context-aware MMLA solutions. In this paper, we identify the requirements and propose a data infrastructure called CIMLA. It includes different data processing components following a standard data processing pipeline and considers contextual information following a data structure. It has been evaluated in three authentic MMLA scenarios involving different cross-disciplinary stakeholders following the Software Architecture Analysis Method. Its fitness was analyzed in each of the three scenarios and developers were interviewed to assess whether it meets functional and non-functional requirements. Results showed that CIMLA supports modularity in developing context-aware MMLA solutions and each of its modules can be reused with required modifications in the development of other solutions. In the future, the current involvement of a developer in customizing the configuration file to consider contextual information can be investigated.

show abstract

Section: Discussionmentioning

confidence: 99%

Section: Existing Mmla Infrastructuresmentioning

confidence: 99%

Section: Tag / Category Requirement Referencesmentioning

confidence: 99%

See 1 more Smart Citation

CIMLA: A Modular and Modifiable Data Preparation, Organization, and Fusion Infrastructure to Partially Support the Development of Context-aware MMLA Solutions

Shankar¹,

Ruiz‐Calleja²,

Prieto³

et al. 2023

jucs

View full text Add to dashboard Cite

show abstract

Building a Robust System for Multimodal Emotion Recognition

Wagner

Lingenfelser

André

2015

Emotion Recognition

View full text Add to dashboard Cite

FILTWAM and Voice Emotion Recognition

Bahreini

Nadolski

Westera

2014

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Abstract. This paper introduces the voice emotion recognition part of our framework for improving learning through webcams and microphones (FILTWAM). This framework enables multimodal emotion recognition of learners during game-based learning. The main goal of this study is to validate the use of microphone data for a real-time and adequate interpretation of vocal expressions into emotional states were the software is calibrated with end users. FILTWAM already incorporates a valid face emotion recognition module and is extended with a voice emotion recognition module. This extension aims to provide relevant and timely feedback based upon learner's vocal intonations. The feedback is expected to enhance learner's awareness of his or her own behavior. Six test persons received the same computer-based tasks in which they were requested to mimic specific vocal expressions. Each test person mimicked 82 emotions, which led to a dataset of 492 emotions. All sessions were recorded on video. An overall accuracy of our software based on the requested emotions and the recognized emotions is a pretty good 74.6% for the emotions happy and neutral emotions; but will be improved for the lower values of an extended set of emotions. In contrast with existing software our solution allows to continuously and unobtrusively monitor learners' intonations and convert these intonations into emotional states. This paves the way for enhancing the quality and efficacy of game-based learning by including the learner's emotional states, and links these to pedagogical scaffolding.

show abstract

The social signal interpretation framework (SSI) for real time signal processing and recognition

Cited by 19 publications

References 12 publications

CIMLA: A Modular and Modifiable Data Preparation, Organization, and Fusion Infrastructure to Partially Support the Development of Context-aware MMLA Solutions

CIMLA: A Modular and Modifiable Data Preparation, Organization, and Fusion Infrastructure to Partially Support the Development of Context-aware MMLA Solutions

Building a Robust System for Multimodal Emotion Recognition

FILTWAM and Voice Emotion Recognition

Contact Info

Product

Resources

About