e-skin: research into wearable interfaces, cross-modal perception and communication for the visually impaired on the mediated stage

Abstract

Today our cultural events are dominated by visual information based on sight and sound, but hardly at all on the combined senses of touch and sound. Visually impaired users, lacking sight are not often able to engage in cultural events. Compensated audio-description or talking books are standard products that visually impaired people can buy to imagine feature film stories or decide what is happening on the stage. Very little theatre, dance or art events exist in which these people can actually participate. Interfaces are not often designed which promote communication between impaired actors, nor allow them to navigate and control audio-visual information on the mediated or digital stage. As neuroscientists suggest, the unique cross-modal potentials of human sensory perception could be augmented by electronic devices, which in turn might communicate with sighted audiences. Our research group is interested to address these problems by constructing ergonomic HCI (Human Computer Interfaces) that can explore the above problems including research into orientation, cognition mapping and external audio-visual device control. Consequently, we are implementing our discoveries in artificial systems, which can interact intelligently with people on the digitally simulated stage.Keywords: HCI, cross-modal interaction, visually impaired users, mediated stages, ergonomic design. The e-skin approach"E-skin" is a set of interfaces, which constitute our past and present attempts to electronically simulate the perceptive modalities of the human skin: pressure, temperature, vibration and proprioception. These four modalities constitute our biggest human organ, constantly detecting and reacting to environmental realities.Our research team is engaged with the mimicry of skin, alongside extensive user testing to assure we make relevant developments for cross-modal potentials, including tactile and acoustic feedback, cognitive mapping and embodied interaction. Cross-modal interaction is the label used in neuroscience to describe how certain features of objects perceived in ones sensory modes go-together with features in another sensory mode.We are particularly interested in the relationship between tactility and acoustic feedback as well as the mediated stage as a feedback loop or "visual substitute". What is the actual relationship between pressure and sound, temperature and volume and proprioception and vibration? Our group explores cognitive mapping, touch, movement, electro-stimulation on the skin itself, gesture and relief differentiation, in order to uncover some answers to this large question. Cognitive mapping exercises or tests about the composition of acquired codes, stores, recalls and decodes, determine relative locations and attributes in the spatial environment of the stage itself.Our mediated stage is a hybrid platform using surround sound and three screens with related potentials of communication and movement and gesture as well as audio-visual response or what theorists call "embodied interaction". As Paul Dour...

Activist Media and Biopolitics: Critical Media Interventions in the Age of Biopower

“…This is not to suggest that this line of research is unimportant. Jill Scott (2008) for example, describes the benefits of e-skin development that enhances touch and sound and enables cross-modal perception to take place through human-computer interfaces (HCI) so that the visually impaired are able to increase their quality of life. Such research may be thought of as the feminine-feminist counterpart to the more cyborgian military fantasies like Jon Favreau's 2008 film, Ironman.…”