PanGEA: The Panoramic Graph Environment Annotation Toolkit

Abstract

PanGEA, the Panoramic Graph Environment Annotation toolkit, is a lightweight toolkit for collecting speech and text annotations in photo-realistic 3D environments. PanGEA immerses annotators in a web-based simulation and allows them to move around easily as they speak and/or listen. It includes database and cloud storage integration, plus utilities for automatically aligning recorded speech with manual transcriptions and the virtual pose of the annotators. Out of the box, PanGEA supports two tasks -collecting navigation instructions and navigation instruction followingand it could be easily adapted for annotating walking tours, finding and labeling landmarks or objects, and similar tasks. We share best practices learned from using PanGEA in a 20,000 hour annotation effort to collect the Room-Across-Room dataset. We hope that our open-source annotation toolkit and insights will both expedite future data collection efforts and spur innovation on the kinds of grounded language tasks such environments can support.

Less is More: Generating Grounded Navigation Instructions from Landmarks

“…We consider human evaluations to be essential for this task. Using PanGEA [36], an opensource annotation toolkit for panoramic graph environments, we immerse annotators in a simulated first-person environment backed by the Matterport3D dataset [9] and ask them to follow the provided text navigation instructions. In total, we conduct 20k wayfinding evaluations involving 70 annotators.…”

Core Challenges in Embodied Vision-Language Planning

“…To do so, the authors provide additional supervision for the agent by dividing R2R instructions into sub-instructions and matching them with visual observations along a the corresponding path. Room-Across-Room (RxR) (Ku et al, 2020) uses the PanGEA annotation tool (Ku et al, 2021) to also tackle path-instruction alignment by densely recording and matching 3D pose traces with audio-based instructions.…”