Multi-modal Open World User Identification

Abstract

User identification is an essential step in creating a personalised long-term interaction with robots. This requires learning the users continuously and incrementally, possibly starting from a state without any known user. In this article, we describe a multi-modal incremental Bayesian network with online learning, which is the first method that can be applied in such scenarios. Face recognition is used as the primary biometric, and it is combined with ancillary information, such as gender, age, height, and time of interaction to improve the recognition. The Multi-modal Long-term User Recognition Dataset is generated to simulate various human-robot interaction (HRI) scenarios and evaluate our approach in comparison to face recognition, soft biometrics, and a state-of-the-art open world recognition method (Extreme Value Machine). The results show that the proposed methods significantly outperform the baselines, with an increase in the identification rate up to 47.9% in open-set and closed-set scenarios, and a significant decrease in long-term recognition performance loss. The proposed models generalise well to new users, provide stability, improve over time, and decrease the bias of face recognition. The models were applied in HRI studies for user recognition, personalised rehabilitation, and customer-oriented service, which showed that they are suitable for long-term HRI in the real world.

Personalised socially assistive robot for cardiac rehabilitation: Critical reflections on long-term interactions in the real world

“…Thus, we applied Multi-modal Incremental Bayesian Network (MMIBN) 1 with online learning (Irfan et al. 2018b , 2021 ), which is the first method for sequential and incremental learning of users that does not require any preliminary training for user recognition. It combines face recognition with soft biometrics , which are ancillary physical or behavioural characteristics, such as gender, age, height and time of interaction, that can be used to improve the recognition performance (Jain et al.…”

“… 2018 ) on a long-term (4 weeks) HRI study in the real world with 14 participants (93.2% identification rate) and on a large artificial multi-modal dataset with 200 users (65.7% identification rate) (Irfan et al. 2018b , 2021 ).…”

“…1 ) to continuously monitor and provide generic feedback to patients during exercise on the treadmill, based on sensory information, or (c) SAR using a NAO robot with personalised features to recognise users (Irfan et al. 2018b , 2021 ), recall the patients’ previous session progress, track their adherence, and give personalised feedback. The results obtained during the first two conditions (Casas et al.…”

“…The multi-modal incremental Bayesian network (Irfan et al. 2018b , 2021 ) for user recognition is available online (see footnote 1). However, other parts of the patient–robot interface are not publicly available.…”

Coffee With a Hint of Data: Towards Using Data-Driven Approaches in Personalised Long-Term Interactions

“…In order to ensure a natural level of interaction with mutual understanding (Mavridis, 2015), non-verbal features, such as gaze (through face tracking) and body movements (i.e., animated speech feature of NAOqi), were used. The interaction was personalised by recognising users with Multi-modal Incremental Bayesian Network (Irfan et al, 2018;Irfan et al, 2021), which combines face recognition with soft biometrics (age, gender, height and time of interaction), and a knowledge-base was used to record and recall user preferences.…”