A sensing architecture for empathetic data systems

Abstract

Today's increasingly large and complex databases require novel and machine aided ways of exploring data. To optimize the selection and presentation of data, we suggest an unconventional approach. Instead of exclusively relying on explicit user input to specify relevant information or to navigate through a data space, we exploit the power and potential of the users' unconscious processes in addition. To this end, the user is immersed in a mixed reality environment while his bodily reactions are captured using unobtrusive wearable devices. The users' reactions are analyzed in real-time and mapped onto higher-level psychological states, such as surprise or boredom, in order to trigger appropriate system responses that direct the users' attention to areas of potential interest in the visualizations. The realization of such a close experience-based human-machine loop raises a number of technical challenges, such as the real-time interpretation of psychological user states. The paper at hand describes a sensing architecture for empathetic data systems that has been developed as part of such a loop and how it tackles the diverse challenges.

Inference of human affective states from psychophysiological measurements extracted under ecologically valid conditions

“…The data coming from the wearable devices is conveyed to the XIM sensing platform that captures and processes in real-time raw sensors data (Wagner et al, 2013a ). This platform is implemented using the Social Signal Interpretation (SSI) framework (Wagner et al, 2013b ), a set of tools for the recording, analysis and recognition of human behavior in real-time.…”

Manipulating complex network structures in virtual reality and 3D printing of the results

“…The user's movements and gestures in the space are tracked using the Microsoft Kinect. In addition, fingers movement and forearm position are tracked using the sensing glove, which was custom-tailored for the XIM space [5]. We use 180 degrees immersive projections to visualize the dataset in 3D.…”

Mindful

“…The works in [14,13] suggest frameworks for interpretation of affective signals. Specifically, in SSI [14] the focus is real-time recognition of affective signals by supporting diverse input modes (e.g., web cameras, Nintendo's Wii, Kinect, Nexus, and more) in a coherent way.…”

“…In addition, the authors provide some machine learning and pattern recognition tools. The CEED [13] utilizes the SSI platform for empathetic data experiments, and augments it by adding affects visualization. The primary focus of these works is on seamless interaction with new sensors through their APIs, and on analysis using machine learning libraries.…”