Recognition and Localisation of Pointing Gestures Using a RGB-D Camera

Lecture Notes in Computer Science

Koutny

Günther

et al. 2020

Self Cite

This paper presents a brainstorming tool combined with pointing gestures to improve the brainstorming meeting experience for blind and visually impaired people (BVIP). In brainstorming meetings, BVIPs are not able to participate in the conversation as well as sighted users because of the unavailability of supporting tools for understanding the explicit and implicit meaning of the non-verbal communication (NVC). Therefore, the proposed system assists BVIP in interpreting pointing gestures which play an important role in non-verbal communication. Our system will help BVIP to access the contents of a Metaplan card, a team member in the brainstorming meeting is referring to by pointing. The prototype of our system shows that targets on the screen a user is pointing at can be detected with 80% accuracy.

Section: Pointing Gesture Recognition Systemmentioning

confidence: 99%

Pointing Gesture Based User Interaction of Tool Supported Brainstorming Meetings

Lecture Notes in Computer Science

Koutny

Günther

et al. 2020

Self Cite

“…One part of the MAPVI project and this software suite is the inclusion of gesture recognition. So far pointing gestures of participants can be recognized [9]. The brainstorming tool allows these pointing gestures to be fed into the system and shared with BVIP.…”

Section: Recognition Of Nvcmentioning

confidence: 99%

Accessible Multimodal Tool Support for Brainstorming Meetings

Koutny

Günther

Lecture Notes in Computer Science

et al. 2020

Self Cite

In recent years, assistive technology and digital accessibility for blind and visually impaired people (BVIP) has been significantly improved. Yet, group discussions, especially in a business context, are still challenging as non-verbal communication (NVC) is often depicted on digital whiteboards, including deictic gestures paired with visual artifacts. However, as NVC heavily relies on the visual perception, whichrepresents a large amount of detail, an adaptive approach is required that identifies the most relevant information for BVIP. Additionally, visual artifacts usually rely on spatial properties such as position, orientation, and dimensions to convey essential information such as hierarchy, cohesion, and importance that is often not accessible to the BVIP. In this paper, we investigate the requirements of BVIP during brainstorming sessions and, based on our findings, provide an accessible multimodal tool that uses non-verbal and spatial cues as an additional layer of information. Further, we contribute by presenting a set of input and output modalities that encode and decode information with respect to the individual demands of BVIP and the requirements of different use cases.

“…Eye gaze provides information on emotional state (Bal et al, 2010), text entry (Majaranta and Räihä, 2007), or concentration for an object (Symons et al, 2004) given by the user, to infer visualization tasks and a user's cognitive abilities (Steichen et al, 2013), to enhance interaction (Hennessey et al, 2014), to have communication via eye gaze patterns (Qvarfordt and Zhai, 2005), etc. However, such information cannot be accessed by blind and visually impaired people (BVIP) as they cannot see where the other person in the meeting room is looking at (Dhingra and Kunz, 2019;Dhingra et al, 2020). Therefore, it is important to track eye gaze in the meeting environment to provide the relevant information to them.…”

Section: Introductionmentioning

confidence: 99%

Eye Gaze Tracking for Detecting Non-verbal Communication in Meeting Environments

Proceedings of the 15th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Application

Hirt

Angst

et al. 2020

Self Cite

Non-verbal communication in a team meeting is important to understand the essence of the conversation. Among other gestures, eye gaze shows the focus of interest on a common workspace and can also be used for an interpersonal synchronisation. If this non-verbal information is missing and or cannot be perceived by blind and visually impaired people (BVIP), they would lack important information to get fully immersed in the meeting and may feel alienated in the course of the discussion. Thus, this paper proposes an automatic system to track where a sighted person is gazing at. We use the open source software 'OpenFace' and develop it as an eye tracker by using a support vector regressor to make it work similarly to commercially available expensive eye trackers. We calibrate OpenFace using a desktop screen with a 2 × 3 box matrix and conduct a user study with 28 users on a big screen (161.7 cm x 99.8 cm x 11.5 cm) with a 1 × 5 box matrix. In this user study, we compare the results of our developed algorithm for OpenFace to an SMI RED 250 eye tracker. The results showed that our work achieved an overall relative accuracy of 58.54%.