Automatic inference of complex affective states

Abstract

Affective states and their non-verbal expressions are an important aspect of human reasoning, communication and social life. Automated recognition of affective states can be integrated into a wide variety of applications for various fields. Therefore, it is of interest to design systems that can infer the affective states of speakers from the non-verbal expressions in speech, occurring in real scenarios. This paper presents such a system and the framework for its design and validation. The framework defines a representation method that comprises a set of affective-state groups or archetypes that often appear in everyday life. The inference system is designed to infer combinations of affective states that can occur simultaneously and whose level of expression can change over time. The framework considers also the validation and generalisation of the system. The system was built of 36 independent pair-wise comparison machines, with average accuracy (tenfold cross validation) of 75%. The accumulated inference system yielded total accuracy of 83% and recognised combinations for different nuances within the affective-state groups. In addition to the ability to recognise these affective-state groups, the inference system was applied to characterisation of a very large variety of affective state concepts (549 concepts) as combinations of the affective-state groups. The system was also applied to annotation of affective states that were naturally evoked during sustained human-computer interactions and multi-modal analysis of the interactions, to new speakers and to a different language, with no additional training. The system provides a powerful tool for recognition, characterisation, annotation (interpretation) and analysis of affective states. In addition, the results inferred from speech in both English and Hebrew, indicate that the vocal expressions of complex affective states such as thinking, certainty and interest transcend language boundaries.

Untitled

“…Many of them have come to recognize that a great deal of human emotional functioning is rooted in unconscious processes. Scientific studies of the T. Sobol-Shikler, for example, offer a deep and useful insight into the conceptual field of this phenomenon by showing that humans pick up the emotional content of unconscious facial expressions that are not intended to influence the perception of others [33]. Van den Noort M. et.al.…”

Comparing animated and static modes in educational gameplay on user interest, performance and gameplay anxiety

“…The term affective states refer to emotions, attitudes, beliefs, intents, desires, and frustration moods. They have cause and effect relations with cognitive processes (Bechara, Damasio, Tranel, & Damasio, 1997) that can be integrated into fields such as human-computer interfaces and interactions (HCI) (Sobol-Shikler, 2011). That is, different HCIs may lead to different levels of frustration, higher levels of maintained interest, and lower levels of gameplay anxiety (Hong et al, 2014).…”

Predicting user mental states in spoken dialogue systems

“…As stated by Sobol-Shikler [12], the term affective state may refer to emotions, attitudes, beliefs, intents, desires, pretending, knowledge and moods.…”

“…Sobol-Shikler [12] shares this vision and proposes a representation method that comprises a set of affectivestate groups or archetypes that often appear in everyday life. His method is designed to infer combinations of affective states that can occur simultaneously and whose level of expression can change over time within a dialogue.…”