We present a classification algorithm for inferring affective states (emotions, mental states, attitudes, and the like) from their nonverbal expressions in speech. It is based on the observations that affective states can occur simultaneously and different sets of vocal features, such as intonation and speech rate, distinguish between nonverbal expressions of different affective states. The input to the inference system was a large set of vocal features and metrics that were extracted from each utterance. The classification algorithm conducted independent pairwise comparisons between nine affective-state groups. The classifier used various subsets of metrics of the vocal features and various classification algorithms for different pairs of affective-state groups. Average classification accuracy of the 36 pairwise machines was 75 percent, using 10-fold cross validation. The comparison results were consolidated into a single ranked list of the nine affective-state groups. This list was the output of the system and represented the inferred combination of co-occurring affective states for the analyzed utterance. The inference accuracy of the combined machine was 83 percent. The system automatically characterized over 500 affective state concepts from the Mind Reading database. The inference of co-occurring affective states was validated by comparing the inferred combinations to the lexical definitions of the labels of the analyzed sentences. The distinguishing capabilities of the system were comparable to human performance.

Affective states and their non-verbal expressions are an important aspect of human reasoning, communication and social life. Automated recognition of affective states can be integrated into a wide variety of applications for various fields. Therefore, it is of interest to design systems that can infer the affective states of speakers from the non-verbal expressions in speech, occurring in real scenarios. This paper presents such a system and the framework for its design and validation. The framework defines a representation method that comprises a set of affective-state groups or archetypes that often appear in everyday life. The inference system is designed to infer combinations of affective states that can occur simultaneously and whose level of expression can change over time. The framework considers also the validation and generalisation of the system. The system was built of 36 independent pair-wise comparison machines, with average accuracy (tenfold cross validation) of 75%. The accumulated inference system yielded total accuracy of 83% and recognised combinations for different nuances within the affective-state groups. In addition to the ability to recognise these affective-state groups, the inference system was applied to characterisation of a very large variety of affective state concepts (549 concepts) as combinations of the affective-state groups. The system was also applied to annotation of affective states that were naturally evoked during sustained human-computer interactions and multi-modal analysis of the interactions, to new speakers and to a different language, with no additional training. The system provides a powerful tool for recognition, characterisation, annotation (interpretation) and analysis of affective states. In addition, the results inferred from speech in both English and Hebrew, indicate that the vocal expressions of complex affective states such as thinking, certainty and interest transcend language boundaries.

No abstract

Human-robot collaboration is essential when the robot operates in unstructured environments which change dynamically and require high perception capabilities. The design of such collaborative systems requires the system to be predictable, i.e., the system's response should be repeatable and as close as possible to the response expected by its users. This should enable users to comprehend and learn how the system operates and foresee the system's responses to their commands and actions. We present a study of remote controlling a robot. This study investigated the expectations, perception, behavior and preferences of users while issuing a "forward" movement command. The study aimed to determine if users expect and prefer a system response that is identical in the distance of movement (repeated quantity), or an adaptive movement whose size depends on the environment (i.e., repeated rule, such as movement until another command, until junctions and/or obstacles). Speech was the only modality of interaction in the two experiments performed. The results show that the movement manner adaptable to the environment is preferred. Although the manner of movement (set step size vs. continuous movement, stopping at junctions vs. not stopping) may affect the overall performance, especially in the learning stage, these differences are not always perceived by the users. Results indicate that a robot's response could be qualitatively similar rather than identical in quantity or quality (the direction is constant, but the movement and feedback manners may vary). Furthermore, the overall gained user experience compensates for minor variations in the system's response.