Looking for Info: Evaluation of Gaze Based Information Retrieval in Augmented Reality

Piening, Robin; Pfeuffer, Ken; Esteves, Augusto; Mittermeier, Tim; Prange, Sarah; Schröder, Philippe; Alt, Florian

doi:10.1007/978-3-030-85623-6_32

Cited by 23 publications

(7 citation statements)

References 44 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…With the advancement of Augmented Reality (AR) devices, researchers have integrated IR closely with physical world through Head-Mounted Displays (HMDs) [3,14,27,39,52,71,76], which is defined as reality-based information retrieval (RBIR) by [15]. RBIR adheres to the principle of "find what you need with zero query terms or less" [7], adopting a proactive strategy to impose information on certain objects in order to reduce inquiry costs [27,39,76].…”

Section: Information Retrieval For Daily Scenariosmentioning

confidence: 99%

“…To tackle information overload in AR, Piening et al [52] proposed gaze-adaptive interfaces to modulate the display of information panels. Meanwhile, researchers [3,14,15] introduced multi-round IR operations leveraging AR's interactivity to facilitate precise information acquisition, including selection and search refinement.…”

Section: Information Retrieval For Daily Scenariosmentioning

confidence: 99%

“…Prior research has investigated the potential of utilizing smart glasses [29,67] equipped with gaze tracking to streamline information retrieval related to physical objects [3,15,26,39,52,71,74]. These approaches enhanced information retrieval by superimposing digital widgets around physical entities, allowing users to access information through interacting with the widgets in a multi-stage manner [3,14].…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

G-VOILA: Gaze-Facilitated Information Querying in Daily Scenarios

Wang,

Shi,

Wang

et al. 2024

Proc. ACM Interact. Mob. Wearable Ubiquitous Technol.

View full text Add to dashboard Cite

Modern information querying systems are progressively incorporating multimodal inputs like vision and audio. However, the integration of gaze --- a modality deeply linked to user intent and increasingly accessible via gaze-tracking wearables --- remains underexplored. This paper introduces a novel gaze-facilitated information querying paradigm, named G-VOILA, which synergizes users' gaze, visual field, and voice-based natural language queries to facilitate a more intuitive querying process. In a user-enactment study involving 21 participants in 3 daily scenarios (p = 21, scene = 3), we revealed the ambiguity in users' query language and a gaze-voice coordination pattern in users' natural query behaviors with G-VOILA. Based on the quantitative and qualitative findings, we developed a design framework for the G-VOILA paradigm, which effectively integrates the gaze data with the in-situ querying context. Then we implemented a G-VOILA proof-of-concept using cutting-edge deep learning techniques. A follow-up user study (p = 16, scene = 2) demonstrates its effectiveness by achieving both higher objective score and subjective score, compared to a baseline without gaze data. We further conducted interviews and provided insights for future gaze-facilitated information querying systems.

show abstract

Section: Information Retrieval For Daily Scenariosmentioning

confidence: 99%

Section: Information Retrieval For Daily Scenariosmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

G-VOILA: Gaze-Facilitated Information Querying in Daily Scenarios

Wang,

Shi,

Wang

et al. 2024

Proc. ACM Interact. Mob. Wearable Ubiquitous Technol.

View full text Add to dashboard Cite

show abstract

“…Even as Augmented Reality (AR) continues to mature technically, it has already become a cutting-edge technique for ordinary use in everyday life , Zhang et al [2021a]. AR features the capacity to superimpose virtual context-related information onto the physical world in the form of graphics Piening et al [2021], , Zhang et al [2021b]. This technology is endorsed and applied in numerous fields due to its capability of providing real-time interaction, as well as generating interactive interfaces of visualized digital content Lee et al [2011], Nowak et al [2021], Zhang et al [2022a].…”

Section: Introductionmentioning

confidence: 99%

“…Human eyes are easy to track, and gaze implicitly conveys what people are interested in. Thus, eye-tracking approaches are becoming more pervasive in interactive AR devices, which are mainly mobile head-mounted or hand-held displays (HMDs and HHDs) Zhang et al [2023, in press], such as Microsoft HoloLens or Magic Leap Piening et al [2021]. Human gaze is utilized for adjusting and adapting virtual information that is being projected onto the real world.…”

Section: Introductionmentioning

confidence: 99%

See or Hear? Exploring the Effect of Visual and Audio Hints and Gaze-assisted Task Feedback for Visual Search Tasks in Augmented Reality

Zhang¹,

Nowak²,

Xuan³

et al. 2023

Preprint

View full text Add to dashboard Cite

Augmented reality (AR) is emerging in visual search tasks for increasingly immersive interactions with virtual objects. We propose an AR approach providing visual and audio hints along with gazeassisted instant post-task feedback for search tasks based on mobile head-mounted display (HMD). The target case was a book-searching task, in which we aimed to explore the effect of the hints together with the task feedback with two hypotheses. H1: Since visual and audio hints can positively affect AR search tasks, the combination outperforms the individuals. H2: The gaze-assisted instant post-task feedback can positively affect AR search tasks. The proof-of-concept was demonstrated by an AR app in HMD and a comprehensive user study (n=96) consisting of two sub-studies, Study I (n=48) without task feedback and Study II (n=48) with task feedback. Following quantitative and qualitative analysis, our results partially verified H1 and completely verified H2, enabling us to conclude that the synthesis of visual and audio hints conditionally improves the AR visual search task efficiency when coupled with task feedback.

show abstract