Semantic awareness through computer vision

Abstract

An important application of multi-user interfaces is distributed presentations. In such presentations, the presenters do not have the ability to assess the real-time level-of-interest of the audience through observation, as they would in real lecture rooms. Using vision techniques, we aim to introduce a path that, if followed, could potentially lead to a robust technique that provides this information in such a presentation in real time.

Engagement Analysis Through Computer Vision

“…Our research is most closely to previous work by Benzaid and Dewan [12] which was the first to automatically determine the engagement level of a remote audience. It did so by analyzing videos of the audience members; specifically it used Viola-Jones [11] facial detection, light image manipulation, and support vector machine (SVM) to classify audience members into three states: Bored, Engaged, and Frustrated.…”

“…The means by which facial data is gathered and classified is relatively straightforward (though a more comprehensive examination can be read in the paper by Benzaid and Dewan [12]), and takes place in several major stages, which are outlined below: Facial data is gathered by means of an arbitrary webcam, in the case of this study, the onboard webcam of a laptop; Any camera capable of providing a basic two-dimensional video feed of users would be viable. During the training phase of the software, a recorded video of such data is reviewed by a user and manually tagged in places where they find themselves to be bored or engaged by the material they were observing.…”

Interactive Control and Visualization of Difficulty Inferences from User-Interface Commands

“…The problems illustrated above are partly addressed by recent mining research on automatically making inferences about users' emotions such as if they are engaged [1], interruptible [4], or in difficulty [2]. Predictions may be made based on characteristics of users (such as their postures [2,7] ) and/or their activities (such as their interaction commands [2,4] ).…”

Semantics-based collaborative systems: Combining the advantages of unstructured and structure-based systems

“…Mining of the actions of software developers has been used to develop awareness mechanisms that try to automatically determine if a software engineer is interruptible [11] or needs help [4,5]. Finally, computer vision techniques have been used to determine if an attendee of a distributed presentation is attentive, bored, or frustrated [3]. Thus, in comparison to both unstructured and structure-based systems, there is less filtering of awareness information to determine the events of interest such as new software conflicts, help need, and change in attention level and interruptibility status.…”