Multi-modal user interface combining eye tracking and hand gesture recognition

Kim, Hansol; Suh, Kun Ha; Lee, Eui Chul

doi:10.1007/s12193-017-0242-2

Cited by 11 publications

(5 citation statements)

References 14 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…If the graphical representation is presented on a large display, as in our case, yet additional challenges arise as concerns how humans interact with it, including window management problems (Robertson et al, 2005). Closer to our interests, not much work exists on interpreting deictic gestures directed to large displays, especially as concerns recognizing the target at a semantic level (Kim et al, 2017).…”

Section: Related Workmentioning

confidence: 98%

Multimodal Coreference Resolution for Exploratory Data Visualization Dialogue: Context-Based Annotation and Gesture Identification

Kumar¹,

Aurisano²,

Eugenio³

et al. 2017

SEMDIAL 2017 (SaarDial) Workshop on the Semantics and Pragmatics of Dialogue

View full text Add to dashboard Cite

The goals of our work are twofold: gain insight into how humans interact with complex data and visualizations thereof in order to make discoveries; and use our findings to develop a dialogue system for exploring data visualizations. Crucial to both goals is understanding and modeling of multimodal referential expressions, in particular those that include deictic gestures. In this paper, we discuss how context information affects the interpretation of requests and their attendant referring expressions in our data. To this end, we have annotated our multimodal dialogue corpus for context and both utterance and gesture information; we have analyzed whether a gesture co-occurs with a specific request or with the context surrounding the request; we have started addressing multimodal co-reference resolution by using Kinect to detect deictic gestures; and we have started identifying themes found in the annotated context, especially in what follows the request.

show abstract

Section: Related Workmentioning

confidence: 98%

Multimodal Coreference Resolution for Exploratory Data Visualization Dialogue: Context-Based Annotation and Gesture Identification

Kumar¹,

Aurisano²,

Eugenio³

et al. 2017

SEMDIAL 2017 (SaarDial) Workshop on the Semantics and Pragmatics of Dialogue

View full text Add to dashboard Cite

show abstract

“…In the 30 years since, not much work has been done on how to accommodate the creation of new entities 5 (see (Wilson et al, 2016) for documents and (Li and Boyer, 2016) for tutoring dialogues about programming), and none in the visualization domain. Note we do not focus on multimodal reference resolution, another vast area (Navarretta, 2011;Qu and Chai, 2008;Eisenstein and Davis, 2006;Prasov and Chai, 2008;Iida et al, 2011;Kim et al, 2017;Sluÿters et al, 2022), even if we will briefly touch on deictic gestures in Section 3.…”

Section: Co-reference Resolutionmentioning

confidence: 99%

Reference Resolution and New Entities in Exploratory Data Visualization: From Controlled to Unconstrained Interactions with a Conversational Assistant

Bhattacharya,

Kumar,

Di Eugenio

et al. 2023

Proceedings of the 24th Meeting of the Special Interest Group on Discourse and Dialogue

View full text Add to dashboard Cite

In the context of data visualization, as in other grounded settings, referents are created by the task the agents engage in and are salient because they belong to the shared physical setting. Our focus is on resolving references to visualizations on large displays; crucially, reference resolution is directly involved in the process of creating new entities, namely new visualizations. First, we developed a reference resolution model for a conversational assistant. We trained the assistant on controlled dialogues for data visualizations involving a single user. Second, we ported the conversational assistant including its reference resolution model to a different domain, supporting two users collaborating on a data exploration task. We explore how the new setting affects reference detection and resolution; we compare the performance in the controlled vs unconstrained setting, and discuss the general lessons that we draw from this adaptation.* Co-first authors displays better support exploration and collaboration (Andrews et al., 2011;Rupprecht et al., 2019;Lischke et al., 2020). In this paper, we focus on new entity establishment via reference in such contexts. We start from the corpus Chicago-Crime-Vis we collected a few years back (Kumar et al., , 2017 in which a user exploring crime data in Chicago interacts with a Visualization Expert (VE) whom they know to be a person generating visualizations on the screen remotely from a separate room. On the basis of Chicago-Crime-Vis, we designed and developed a version of our assistant which was called Articulate2 Kumar et al., 2020) 1 . We will report the performance of Articulate2 on reference resolution, and especially reference establishment, with respect to the transcribed and annotated Chicago-Crime-Vis corpus, evaluated in an offline manner. The second part of our paper discusses the challenges that arose when we ported Articulate2 to a new setting: two collaborators work together to assess COVID policies given geographic and demographic features of the data, and interact exclusively with the deployed Articulate+ (see Figure 1). We will illustrate the many issues which degrade performance, from speech processing errors, to the adaptation of models to new domains, to the inherently more complex setting in which the assistant is now behaving like an overhearer of somebody else's conversations. For clarity, we will refer to Articulate2 in the city crime domain as Art-City-Asst, and to Articulate+ in the COVID domain, as Art-COVID-Asst.A disclaimer before we proceed: the purpose Chicago-Crime-Vis (H) COVID (A) COVID (T)

show abstract

“…Likewise, an eyetracker combined with haptic feedback in virtual reality-based games is proposed for learning-based games [21]. Multimodal systems are advanced for communique functions, combining eye-tracking, gesture, and contact-and-voice input [22], [23], [24], [25], however, they did not consider potential end users in their studies. Also, it is unclear how the choice of the modality can impact the performance, especially if children with dyslexia have a different performance.…”

Section: Introductionmentioning

confidence: 99%

Detection of Dyslexic Children Using Machine Learning and Multimodal Hindi Language Eye-Gaze-Assisted Learning System

Meena

Cecotti

Bhushan

et al. 2023

IEEE Trans. Human-Mach. Syst.

View full text Add to dashboard Cite

Children with dyslexia need specific instructions for spelling and word analysis from an early age. It is important to provide appropriate tools using technology for writing aids to such children that can help them to input text, while providing multiple feedback. However, it is unclear how children with dyslexia can efficiently use a gaze-based virtual keyboard. In this study, we propose to use the typing performance of a multimodal Hindi language eye-gaze assisted learning system based on a virtual keyboard to help in the reduction of tracking errors for people with writing and reading deficiencies and to detect children with dyslexia. Performance was assessed at three levels: eye-tracker, eye-tracker with soft-switch, and touchscreen as a baseline modality using a predefined copy-typing task. The system was validated through a series of experiments with 32 children (16 dyslexic and 16 control). The results show that the workload and the usability of the system are substantially different for children with dyslexia. Children with dyslexia have a lower typing performance when using the touchscreen modality or the eye-tracker only. The detection of children with dyslexia from others was assessed with seven different types of classifiers using the typing speed on different words (AUC>0.9). These results highlight the need to have fully inclusive virtual keyboards. This work demonstrates the superior use of a multimodal system with participants having unique neuropsychological conditions and that the proposed system can be used to detect children with dyslexia.

show abstract

Multi-modal user interface combining eye tracking and hand gesture recognition

Cited by 11 publications

References 14 publications

Multimodal Coreference Resolution for Exploratory Data Visualization Dialogue: Context-Based Annotation and Gesture Identification

Multimodal Coreference Resolution for Exploratory Data Visualization Dialogue: Context-Based Annotation and Gesture Identification

Reference Resolution and New Entities in Exploratory Data Visualization: From Controlled to Unconstrained Interactions with a Conversational Assistant

Detection of Dyslexic Children Using Machine Learning and Multimodal Hindi Language Eye-Gaze-Assisted Learning System

Contact Info

Product

Resources

About