Mixing realities for sketch retrieval in Virtual Reality

Abstract

Drawing tools for Virtual Reality (VR) enable users to model 3D designs from within the virtual environment itself. These tools employ sketching and sculpting techniques known from desktopbased interfaces and apply them to hand-based controller interaction. While these techniques allow for mid-air sketching of basic shapes, it remains difficult for users to create detailed and comprehensive 3D models. In our work, we focus on supporting the user in designing the virtual environment around them by enhancing sketch-based interfaces with a supporting system for interactive model retrieval. Through sketching, an immersed user can query a database containing detailed 3D models and replace them into the virtual environment. To understand supportive sketching within a virtual environment, we compare different methods of sketch interaction, i.e., 3D mid-air sketching, 2D sketching on a virtual tablet, 2D sketching on a fixed virtual whiteboard, and 2D sketching on a real tablet. Our results show that 3D mid-air sketching is considered to be a more intuitive method to search a collection of models while the addition of physical devices creates confusion due to the complications of their inclusion within a virtual environment. While we pose our work as a retrieval problem for 3D models of chairs, our results can be extrapolated to other sketching tasks for virtual environments.

Mixing Modalities of 3D Sketching and Speech for Interactive Model Retrieval in Virtual Reality

“…They gather handdrawn 3D sketches and generate a dataset paired with a 3D model collection. Giunchi et al [18,20,21] implemented a VR application that uses sketch as the main mechanism to search a target chair among an extensive collection. This software provides the user with an iterative method of depicting a chair, querying a neural network with a 3D sketch at each step, or by fusing the sketch and a chair model.…”

“…In contrast, 3D sketch as query-by-example is an under-explored research area in the context of virtual environments. Recent works have shown that 3D sketch alone does not lead to high accuracy results [18]. Giunchi et al [19] address this issue by presenting the user with a navigable set of query results instead of a single one.…”

3D sketching for interactive model retrieval in virtual reality

“…They gather hand-drawn 3D sketches and generate a dataset paired with a 3D model collection. Giunchi et al [18,20,21] implemented a VR application that uses sketch as the main mechanism to search a target chair among an extensive collection. This software provides the user with an iterative method of depicting a chair, querying a neural network with a 3D sketch at each step, or by fusing the sketch and a chair model.…”

“…In contrast, 3D sketch as query-by-example is an under-explored research area in the context of virtual environments. Recent works have shown that 3D sketch alone does not lead to high accuracy results [18]. Giunchi et al [19] address this issue by presenting the user with a navigable set of query results instead of a single one.…”