Conveying facial expressions to blind and visually impaired persons through a wearable vibrotactile device

Buimer, Hendrik; Bittner, Marian; Kostelijk, Tjerk; Geest, Thea M. van der; Nemri, Abdellatif; Wezel, R.J.A. van; Zhao, Yan

doi:10.1371/journal.pone.0194737

Cited by 33 publications

(19 citation statements)

References 35 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Additionally, various data mining and machine learning techniques could be applied to set up models for an individual's emotional profile based on sensor-based physiological and behavioral recordings. This could further be transferred to various positive computing use-cases 88 , such as helping children with autism in their social communication 89,90 , helping people who are blind to read facial expressions and get the emotion information of their peers 91 , finding opportune moments for conversational user interactions 92,93 , assisting social anxiety disorder patients to overcome their conditions 94 , allowing robots to interact more intelligently with people 95,96 , and monitoring signs of frustration and emotional saturation that affect attention while driving, to enhance driver safety 97,98 .…”

Section: Usage Notesmentioning

confidence: 99%

K-EmoCon, a multimodal sensor dataset for continuous emotion recognition in naturalistic conversations

et al. 2020

View full text Add to dashboard Cite

Recognizing emotions during social interactions has many potential applications with the popularization of low-cost mobile sensors, but a challenge remains with the lack of naturalistic affective interaction data. Most existing emotion datasets do not support studying idiosyncratic emotions arising in the wild as they were collected in constrained environments. Therefore, studying emotions in the context of social interactions requires a novel dataset, and K-EmoCon is such a multimodal dataset with comprehensive annotations of continuous emotions during naturalistic conversations. The dataset contains multimodal measurements, including audiovisual recordings, EEG, and peripheral physiological signals, acquired with off-the-shelf devices from 16 sessions of approximately 10-minute long paired debates on a social issue. Distinct from previous datasets, it includes emotion annotations from all three available perspectives: self, debate partner, and external observers. Raters annotated emotional displays at intervals of every 5 seconds while viewing the debate footage, in terms of arousal-valence and 18 additional categorical emotions. The resulting K-EmoCon is the first publicly available emotion dataset accommodating the multiperspective assessment of emotions during social interactions.

show abstract

Section: Usage Notesmentioning

confidence: 99%

K-EmoCon, a multimodal sensor dataset for continuous emotion recognition in naturalistic conversations

et al. 2020

View full text Add to dashboard Cite

show abstract

“…Many assistive technologies for blind people has traditionally focused on mobility, navigation, and object recognition; but more recently on social interaction as well [24], [25]. An increasing number of studies explored to assist blind people in social situations based on smart technologies, such as identify faces and facial expressions of the sighted counterparts [26], [7]- [28]. A facial recognition system can help blind people to identify colleagues in group meetings [27].…”

Section: B Social Signal Perception and Technologymentioning

confidence: 99%

“…In the HFD system, 48 vibration motors are mounted on the back of a chair to best map the facial movements and the corresponding vibration cues. Buimer et al [26] introduced a sensory substitution device (SSD) to support blind people to determine facial expressions of their interaction partners. The SSD classifies six universal facial expressions [30] into emotions, conveying vibrotactile stimuli by a belt.…”

Section: Instead Of Delivering Information Through Sense Of Hearingmentioning

confidence: 99%

An Evaluation of a Wearable Assistive Device for Augmenting Social Interactions

Qiu

Han

et al. 2020

IEEE Access

View full text Add to dashboard Cite

Gaze behaviors contain rich information regarding a person's emotions and engagements. Reciprocal eye contact can invoke feelings of liking between two strangers. But blind people cannot perceive and establish the eye contact with sighted counterparts, causing their feelings of social isolation and low confidence in conversations. Thus, our research purpose is to let blind people perceive and react gaze behaviors in social interactions. A Social Glasses system has been implemented iteratively to deliver the multisensory feedback channels of the "eye contact", integrating both visual and tactile feedback. Specifically, the system consists of a Social Glasses device and a tactile wristband, which are worn by a blind person. The Social Glasses simulates the natural gaze for the blind person, aiming at establishing the "eye contact" between blind and sighted people. The tactile wristband enables the blind person to perceive the corresponding tactile feedback when an "eye contact" happens. To test the system, we conducted a user experiment with 40 participants, including 10 blind-sighted pairs (N=20) and 10 blindfolded-sighted pairs (N=20), to see how it could help increase the communication quality between blind and sighted people, as well as to suggest implications for its design. Our main findings demonstrated that both the simulated gaze and the tactile feedback were significantly effective to enhance the communication quality in blind-sighted conversations. Overall, we contribute (1) empirical research findings on how a Social Glasses system enhances the communication quality in blind-sighted conversations; and (2) design principles to inform future assistive wearable device for augmenting social interactions. INDEX TERMS Accessibility, artificial gaze, assistive technology, communication quality, eye tracking, face-to-face communication, gaze contact, visual impairments.

show abstract

“…Use Case Emotions. Persons with visual impairments often have difficulties perceiving nonverbal conversational clues, such as facial expressions [2], making social interactions challenging. To support a user during a conversation, an assistive system has to be able to first extract and process these clues and then convey the information in an adapted format to the user.…”

Section: Related Workmentioning

confidence: 99%

“…A neural network can provide more robustness to image properties or fuse multiple modalities [4]. For presenting the information, several researchers have proposed solutions for people with VI using audio [1] or tactile [2] feedback. Zuniga et al proposed a system providing visual feedback [14].…”

Section: Related Workmentioning

confidence: 99%

Augmented Reality for People with Low Vision: Symbolic and Alphanumeric Representation of Information

Läng

Schmidt

Machulla

2020

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Many individuals with visual impairments have residual vision that often remains underused by assistive technologies. Head-mounted augmented reality (AR) devices can provide assistance, by recoding difficult-to-perceive information into a visual format that is more accessible. Here, we evaluate symbolic and alphanumeric information representations for their efficiency and usability in two prototypical AR applications: namely, recognizing facial expressions of conversational partners and reading the time. We find that while AR provides a general benefit, the complexity of the visual representations has to be matched to the user’s visual acuity.

show abstract

Conveying facial expressions to blind and visually impaired persons through a wearable vibrotactile device

Cited by 33 publications

References 35 publications

K-EmoCon, a multimodal sensor dataset for continuous emotion recognition in naturalistic conversations

K-EmoCon, a multimodal sensor dataset for continuous emotion recognition in naturalistic conversations

An Evaluation of a Wearable Assistive Device for Augmenting Social Interactions

Augmented Reality for People with Low Vision: Symbolic and Alphanumeric Representation of Information

Contact Info

Product

Resources

About