AssistQ: Affordance-Centric Question-Driven Task Completion for Egocentric Assistant

Wong, Benita; Chen, Joya; Wu, You; Lei, Stan Weixian; Mao, Dongxing; Gao, Difei; Shou, Mike Zheng

doi:10.1007/978-3-031-20059-5_28

Cited by 11 publications

(4 citation statements)

References 25 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The same work presented an extensive analysis of several VQA methods highlighting the effective support provided by large language models. Wong et al (2022) defined the affordance-centric VQA problem where the AI assistant should learn from instructional videos to provide step-by-step help in the user's view. The authors introduced a new dataset and developed a novel question-to-action model based on an encoder-decoder architecture.…”

Section: Visual Question Answering (Vqa)mentioning

confidence: 99%

“…The QA task is formulated as a classification problem over the whole answer vocabulary. The AQTC benchmark proposed by Wong et al (2022) was created with a close focus on task completion and affordances. It contains 100 instructional videos with an average duration of 115 seconds and involves 25 common household appliances, with 531 multiple-choice question-answer samples.…”

Section: Visual Question Answering (Vqa)mentioning

confidence: 99%

See 1 more Smart Citation

An Outlook into the Future of Egocentric Vision

Plizzari,

Goletto,

Furnari

et al. 2024

Int J Comput Vis

View full text Add to dashboard Cite

What will the future be? We wonder! In this survey, we explore the gap between current research in egocentric vision and the ever-anticipated future, where wearable computing, with outward facing cameras and digital overlays, is expected to be integrated in our every day lives. To understand this gap, the article starts by envisaging the future through character-based stories, showcasing through examples the limitations of current technology. We then provide a mapping between this future and previously defined research tasks. For each task, we survey its seminal works, current state-of-the-art methodologies and available datasets, then reflect on shortcomings that limit its applicability to future research. Note that this survey focuses on software models for egocentric vision, independent of any specific hardware. The paper concludes with recommendations for areas of immediate explorations so as to unlock our path to the future always-on, personalised and life-enhancing egocentric vision.

show abstract

Section: Visual Question Answering (Vqa)mentioning

confidence: 99%

Section: Visual Question Answering (Vqa)mentioning

confidence: 99%

An Outlook into the Future of Egocentric Vision

Plizzari,

Goletto,

Furnari

et al. 2024

Int J Comput Vis

View full text Add to dashboard Cite

show abstract

“…Interactive learning could be applicable to many applications in the metaverse, in which humans interact with the model in both virtual and physical worlds, convey and discover new knowledge. As a starting point, recent works [182], [183] introduce AI assistants on smart glasses that can instruct novices in using a new device or learning a new skill. Yet, these models still do not form a bidirectional feedback loop between users and learning systems.…”

Section: J Ar/vr Data Streaming and Learningmentioning

confidence: 99%

Managing Metaverse Data Tsunami: Actionable Insights

Zhang,

Chen,

Ooi

et al. 2024

IEEE Trans. Knowl. Data Eng.

View full text Add to dashboard Cite

In the metaverse the physical space and the virtual space co-exist, and interact simultaneously. While the physical space is virtually enhanced with information, the virtual space is continuously refreshed with real-time, real-world information.To allow users to process and manipulate information seamlessly between the real and digital spaces, novel technologies must be developed. These include smart interfaces, new augmented realities, and efficient data storage, management, and dissemination techniques. In this paper, we first discuss some promising cospace applications. These applications offer opportunities that neither of the spaces can realize on its own. Then, we further discuss several emerging technologies that empower the construction of metaverse. After that, we discuss comprehensively the data centric challenges. Finally, we discuss and envision what are likely to be required from the database and system perspectives.

show abstract

“…Egocentric videos have become popular in the computer vision community due to the prevalence of small cameras and the ease of collecting FPV videos. These cameras are useful to collect a variety of videos in different domains, such as manufacturing (assemblydisassembly) (Tan et al, 2020), education, behavior (Bambach et al, 2017;Wong et al, 2022), and sports . These videos also enable analysis such as object detection (Fathi et al, 2011;Fan et al, 2018;Furnari et al, 2017), hands detection (Bambach et al, 2015;Chen et al, 2023), gaze detection and prediction (Huang et al, 2018).…”

Section: Video Qamentioning

confidence: 99%

GazeVQA: A Video Question Answering Dataset for Multiview Eye-Gaze Task-Oriented Collaborations

Ilaslan,

Song,

Chen

et al. 2023

Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing

View full text Add to dashboard Cite

The usage of exocentric and egocentric videos in Video Question Answering (VQA) is a new endeavor in human-robot interaction and collaboration studies. Particularly for egocentric videos, one may leverage eye-gaze information to understand human intentions during the task. In this paper, we build a novel task-oriented VQA dataset, called GazeVQA, for collaborative tasks where gaze information is captured during the task process. GazeVQA is designed with a novel QA format that covers thirteen different reasoning types to capture multiple aspects of task information and user intent. For each participant, GazeVQA consists of more than 1,100 textual questions and more than 500 labeled images that were annotated with the assistance of the Segment Anything Model. In total, 2,967 video clips, 12,491 labeled images, and 25,040 questions from 22 participants were included in the dataset. Additionally, inspired by the assisting models and common ground theory for industrial task collaboration, we propose a new AI model called AssistGaze that is designed to answer the questions with three different answer types, namely textual, image, and video. AssistGaze can effectively ground the perceptual input into semantic information while reducing ambiguities. We conduct comprehensive experiments to demonstrate the challenges of GazeVQA 1 and the effectiveness of AssistGaze 2 . * Equal contribution.

show abstract

AssistQ: Affordance-Centric Question-Driven Task Completion for Egocentric Assistant

Cited by 11 publications

References 25 publications

An Outlook into the Future of Egocentric Vision

An Outlook into the Future of Egocentric Vision

Managing Metaverse Data Tsunami: Actionable Insights

GazeVQA: A Video Question Answering Dataset for Multiview Eye-Gaze Task-Oriented Collaborations

Contact Info

Product

Resources

About