BimodalGaze: Seamlessly Refined Pointing with Gaze and Filtered Gestural Head Movement

Sidenmark, Ludwig; Mardanbegi, Diako; Gómez, Argenis Ramírez; Clarke, Christopher; Gellersen, Hans

doi:10.1145/3379155.3391312

Cited by 34 publications

(46 citation statements)

References 32 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Most of the previous studies only supported simple tasks such as pointing and selection by combining eye gazing and head gestures [23], [24], [34]- [36]. Sidenmark and Gellersen [23] proposed leveraging the synergetic movement of eye and head and identified design principles for Eye&Head gaze interaction.…”

Section: A Eye Gaze and Head Movement-based Interactions In Pointing And Selectionmentioning

confidence: 99%

“…Recently, several studies were conducted to utilize eye gazing and head gestures for hands-free interactions because eye-tracking sensors and gyro-sensors are embedded in smart devices. Sidenmark and Gellersen [23], Sidenmark et al [24] leveraged the synergetic movement of eye and head with naturally combined eye-head movement and refined the cursor position with gestural head movement. However, they performed only simple tasks such as selection, and they did not perform more complicated tasks such as 3D manipulation with 3D virtual objects.…”

Section: Introductionmentioning

confidence: 99%

“…Other studies focused on controlling wheelchairs and robots with head gestures for disabled people who could not use their hands [18]- [21]. It is important to note that eye gazing causes less fatigue than head gesturing because the eye gaze can move quickly without head movement, but the accuracy is lower than that of the head movement due to the jittering of the eye movement and the limitation of the performance of the eye tracker [24], [25]. For this reason, when the robot is directly manipulated using eye gazing or head gestures, it is very difficult to control its position and orientation.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Hands-Free Human–Robot Interaction Using Multimodal Gestures and Deep Learning in Wearable Mixed Reality

et al. 2021

View full text Add to dashboard Cite

This study proposes a novel hands-free interaction method using multimodal gestures such as eye gazing and head gestures and deep learning for human-robot interaction (HRI) in mixed reality (MR) environments. Since human operators hold some objects for conducting tasks, there are many constrained situations where they cannot use their hands for HRI interactions. To provide more effective and intuitive task assistance, the proposed hands-free method supports coarse-to-fine interactions. Eye gazing-based interaction is used for coarse interactions such as searching and previewing of target objects, and head gesture interactions are used for fine interactions such as selection and 3D manipulation. In addition, deep learning-based object detection is applied to estimate the initial positioning of physical objects to be manipulated by the robot. The result of object detection is then combined with 3D spatial mapping in the MR environment for supporting accurate initial object positioning. Furthermore, virtual object-based indirect manipulation is proposed to support more intuitive and efficient control of the robot, compared with traditional direct manipulation (e.g., joint-based and end effector-based manipulations). In particular, a digital twin, the synchronized virtual robot of the real robot, is used to provide a preview and simulation of the real robot to manipulate it more effectively and accurately. Two case studies were conducted to confirm the originality and advantages of the proposed hands-free HRI: (1) performance evaluation of initial object positioning and (2) comparative analysis with traditional direct robot manipulations. The deep learningbased initial positioning reduces much effort for robot manipulation using eye gazing and head gestures. The object-based indirect manipulation also supports more effective HRI than previous direct interaction methods. INDEX TERMS Deep learning, eye gazing, hands-free interaction, head gestures, human-robot interaction, mixed reality, object detection.

show abstract

Section: A Eye Gaze and Head Movement-based Interactions In Pointing And Selectionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Hands-Free Human–Robot Interaction Using Multimodal Gestures and Deep Learning in Wearable Mixed Reality

et al. 2021

View full text Add to dashboard Cite

show abstract

“…Both rigid shapes (e.g., rectangular [107]) or circular [108] and flexible shapes (e.g., [109]) have been used, as well as various display media (e.g., projection on cardboard [17,107]), transparent props [12,98], handheld touchscreens [40,59], or virtual lenses [64,82]. In addition, the combination of eye-gaze with other modalities such as touch [85,86], mid-air gestures [87,97,101] and head-movements [56,103,104] has been recently investigated for interaction in spatial user interfaces. For a recent survey on gaze-based interaction in AR and VR, see Hirzle et al [47].…”

Section: Spatial Interactionmentioning

confidence: 99%

Breaking the Screen: Interaction Across Touchscreen Boundaries in Virtual Reality for Mobile Knowledge Workers

Biener

Schneider

Gesslein

et al. 2020

IEEE Trans. Visual. Comput. Graphics

View full text Add to dashboard Cite

Virtual Reality (VR) has the potential to transform knowledge work. One advantage of VR knowledge work is that it allows extending 2D displays into the third dimension, enabling new operations, such as selecting overlapping objects or displaying additional layers of information. On the other hand, mobile knowledge workers often work on established mobile devices, such as tablets, limiting interaction with those devices to a small input space. This challenge of a constrained input space is intensified in situations when VR knowledge work is situated in cramped environments, such as airplanes and touchdown spaces. In this paper, we investigate the feasibility of interacting jointly between an immersive VR head-mounted display and a tablet within the context of knowledge work. Specifically, we 1) design, implement and study how to interact with information that reaches beyond a single physical touchscreen in VR; 2) design and evaluate a set of interaction concepts; and 3) build example applications and gather user feedback on those applications.

show abstract

“…Pursuits are also suitable for hands-free object selections in virtual environments [7], or even selecting occluded objects [21]. Recent literature also gives insights on utilizing eye gaze compensated with head movements for gaze-based selections for improving targeting precision [10], enhancing usability [11,23], and freeing eye gaze exploration [22].…”

Section: Eye Tracking and Controlmentioning

confidence: 99%

Eye Gaze-based Object Rotation for Head-mounted Displays

Liu

Orlosky

Plopski

2020

Symposium on Spatial User Interaction

View full text Add to dashboard Cite

Figure 1: Images showing the three methods we developed to handle eye gaze-based rotations of virtual objects on headmounted displays. a) The initial interface for selecting an axis by which to rotate. b)-d) Examples of RotBar, which maps per-axis rotations onto one-dimensional eye gaze movements. e)-g) Examples of RotPlane, which makes use of orthogonal planes to achieve per-axis angular rotations. h) i) Examples of RotBall, which combines a traditional arcball with a ring for handling user-perspective roll (z-axis) rotations. The yellow circle highlights the eye gaze point in the figures, but is not rendered during use.

show abstract

BimodalGaze: Seamlessly Refined Pointing with Gaze and Filtered Gestural Head Movement

Cited by 34 publications

References 32 publications

Hands-Free Human–Robot Interaction Using Multimodal Gestures and Deep Learning in Wearable Mixed Reality

Hands-Free Human–Robot Interaction Using Multimodal Gestures and Deep Learning in Wearable Mixed Reality

Breaking the Screen: Interaction Across Touchscreen Boundaries in Virtual Reality for Mobile Knowledge Workers

Eye Gaze-based Object Rotation for Head-mounted Displays

Contact Info

Product

Resources

About