Applications of the Multimodal Interaction Architecture in Ambient Assisted Living

The AM4I Architecture and Framework for Multimodal Interaction and Its Application to Smart Environments

“…Despite its focus on web scenarios, and some open questions, such as how to accomplish the discovery of new modalities available in the system, it seems as a good starting point to create multimodal applications as it serves the principles of a decoupled architecture, enabling easy integration of modalities, and is open enough to encompass new evolutions. These standards have inspired the proposal of frameworks such as OpenStream Cue-me () and have also been considered in our work when we extended our agent-based architecture for Ambient Assisted Living (AAL) [15,16,17].…”

“…The definition of the architecture, considering the aforementioned requirements, profits from our previous work in designing an architecture to address Ambient Assisted Living scenarios (AAL) with a single end-user application for desktop and mobile devices [15]. For that context, our proposal was aligned with the Multimodal Interaction Architecture proposed by the W3C [13], but going beyond the target of web contexts specified in the standard.…”

“…The described architecture includes modules to support A daptive, M ultilingual, M ultiplatform, M ultidevice, M ultimodal I nteraction, ( AM4I , “Amphorae”, for short), as follows: Multimodal Interaction —The architecture supports a variable and dynamic number of input and output modalities by adopting an Interaction Manager assisted by Fusion and Fission services. These modules, together, can be seen as a macro Interaction Manager capable of receiving events from input (and passive) modalities, combining them (considering, for instance, a CARE model [5,56]) and sending information to output modalities.In the first approach, each device runs an instance of the Interaction Manager while, in the second approach, a single cloud based Interaction Manager runs remotely and each device connects to it. Multidevice —The proposal encompasses Interaction Managers, which are continuously running at building servers or the cloud, and all the interaction modalities know their location and communicate with them [15,17]. Complex devices, such as smartphones, integrate in the architecture manifold: as platforms to run applications, as (several) passive modalities, and providing multiple input and output modalities. Multiplatform —One important characteristic of the architecture, originating from its decoupled, distributed nature, is the ability to encompass multiple platforms.…”

“…Multidevice —The proposal encompasses Interaction Managers, which are continuously running at building servers or the cloud, and all the interaction modalities know their location and communicate with them [15,17]. Complex devices, such as smartphones, integrate in the architecture manifold: as platforms to run applications, as (several) passive modalities, and providing multiple input and output modalities.…”

“…To be of use, the architecture presented in the previous section needs to be instantiated. Extending previous work for multimodal interaction in AAL scenarios [15,16] the AM4I framework has been developed, enabling concrete deployment of multimodal interactive capabilities in Smart Environments.…”