Multimodal Interaction Framework Based on Firebase Real-Time Database

Abstract

Relying on one technology with a single interaction modality may benefit some users but would certainly exclude a lot more if they have impedances to use that modality. The solution then becomes the inclusion of multiple modalities in the initial design of the interactive system making it more adaptable to the needs of many more users. Including many modalities can rapidly increase the number of interaction objects that need to receive the stream of user commands. This is especially true if the user needs to interact with multiple artifacts in a home automation environment. In this paper, we present the general architecture of an ongoing project for multimodal home automation system. This system relies on a web based database called Firebase for the exchange of user input and the issuing of commands to the multiple artifacts. The user input is acquired using a smartphone and a webcam equipped computer. They capture the user's tactile input, vocal phrases, eye gaze as well as head pose features like tilt and face direction. We were able to achieve a reliable data transfer between the database and the different input acquisition interface. As a first step in the prototyping of the system, we were able to control two separate game interfaces developed using Unity3D software.

Multimodal Interaction for Cobot Using MQTT

“…and would make a final decision to send these to the robot via a single MQTT message, after having dealt with any possible conflicts [44]. This kind of solution was presented in [2] for a paint application using Firebase and App Inventor (see multimodal engine on Figure 1), and is certainly reproducible with MQTT.…”

“…App Inventor for Android is an application development software created by Google, currently maintained by the Massachusetts Institute of Technology [30]. We successfully used App Inventor and an experimental component named "Firebase DB", in previous work [2] in order to create quickly and easily some mobile applications to deploy and test multimodal prototypes.…”

“…In previous works, we have introduced a Multimodal Interaction Framework based on Firebase Realtime Database [2] and a Biomedical Framework for Enhanced Experimentation -BIOFEE [3]. The interest of such intelligent systems built on a multimodal basis lies in the fact that a decision is made through several independent information channels, with the subsequent aggregation of these decisions [4].…”

BIOFEE: Biomedical Framework for Enhanced Experimentation

“…-Health situation and mental state changing from one session to another; -Calibration and sharp synchronization of sensors and effectors needed; -Requirement to test various input and output solutions... In a previous work, we have introduced a Multimodal Interaction Framework based on Firebase Real-Time Database (Guedira and Rouillard 2021). The interest of such intelligent systems built on a multimodal basis lies in the fact that a decision is made through several independent information channels with the subsequent aggregation of these decisions (Filist et al 2022).…”