Modelling User Affect and Sentiment in Intelligent User Interfaces

Abstract

The computer-based automatic analysis of human sentiment, and affect are broadly expected to play a major role that will likely make 'that difference' in future Intelligent User Interfaces, as they bear the promise to lend interactive systems emotional intelligence. Such comprise intelligent digital games, e. g., for empowerment and inclusion, tutoring systems, information systems or virtual companions, e. g., in the car to name but a few. This tutorial aims to give a good introduction into the related fields of user Sentiment Analysis and user Affect Modelling. Its intention is to show the general technology, and its current reliability, the ways for technical integration and efficient embedding of solutions in a user interface context, and the latest trends in this young and ever emerging field. Emphasis is laid on highlighting the range of toolkits available at this moment with the aim of empowering one to immediately craft own solutions. This description contains the general motivation,

Introduction to Intelligent User Interfaces (IUIs)

“…None the less, to get to the stage where we can provide users with virtual interfaces, there is always a long way to go. Computer-based automated interpretation of human sentiment and effect is generally predicted to play a significant role in future Intelligent User Interfaces, as it carries the potential of supplying immersive applications with emotional intelligence [48].…”

Cultural heritage visits supported on visitors’ preferences and mobile devices

“…Schuller [49] argues that what will differentiate future IUI is the commitment to lend them "emotional intelligence": interfaces that realize and can react properly to the users' pleasure or displeasure. The information is thereby increasingly accessed from multiple modalities, in affect recognition and sentiment analysis, thanks to the availability of increasingly large and realistic resources, including deep learning and long short-term memory architectures, and weakly supervised learning methods [36,39,49]. In fact, in the best of all worlds, the system would have sufficient knowledge about a users' culture before their first accesses to the interface, because the first impression counts [34].…”

An Initial Framework to Develop a Mobile Five Human Senses Augmented Reality System for Museums

“…Akiki, Bandara, and Yu (2014) presented a study about adaptive model-driven UI development systems, where the focus was AUI for mobile application applied to museums, e.g., MNEMOSYNE (Karaman et al, 2016). Schuller (2015) mentioned that what differences future IUIs is the promise to lend them "emotional intelligence": interfaces that "know" and can react appropriately to the satisfaction or anger of their users. The information is thereby increasingly accessed from multiple modalities (Morency, Mihalcea, & Doshi, 2011), in affect recognition and sentiment analysis, thanks to the availability of increasingly large and realistic resources (Schuller, 2015), including deep learning and long-short-term memory architectures (Metallinou et al, 2012), and weakly supervised learning methods.…”

“…Schuller (2015) mentioned that what differences future IUIs is the promise to lend them "emotional intelligence": interfaces that "know" and can react appropriately to the satisfaction or anger of their users. The information is thereby increasingly accessed from multiple modalities (Morency, Mihalcea, & Doshi, 2011), in affect recognition and sentiment analysis, thanks to the availability of increasingly large and realistic resources (Schuller, 2015), including deep learning and long-short-term memory architectures (Metallinou et al, 2012), and weakly supervised learning methods. In fact, in the best of all worlds, the system would have sufficient knowledge about a user's culture before he/she first accesses the interface, because the first impression counts (Lindgaard, Fernandes, Dudek, & Brown, 2006).…”