Spatial Perception Enhancement in Assembly Training Using Augmented Volumetric Playback

Sasikumar, Prasanth; Chittajallu, Soumith; Raj, Navindd; Bai, Huidong; Billinghurst, Mark

doi:10.3389/frvir.2021.698523

Cited by 9 publications

(1 citation statement)

References 40 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Therefore, mixed reality can significantly improve performance and user experience in many remote collaboration scenarios. In some complex production scenarios, such as assembly [ 4 , 5 ], emergency maintenance [ 6 , 7 ] and training [ 8 , 9 ], local operators often face problems when assembling or disassembling parts due to their limited domain knowledge. They lack the experience to identify the causes of the problems and do not know how to solve them correctly and efficiently.…”

Section: Introductionmentioning

confidence: 99%

Can you notice my attention? A novel information vision enhancement method in MR remote collaborative assembly

Yan

Bai

et al. 2023

Int J Adv Manuf Technol

View full text Add to dashboard Cite

In mixed reality (MR) remote collaborative assembly, remote experts can guide local users to complete the assembly of physical tasks by sharing user cues (eye gazes, gestures, etc.) and spatial visual cues (such as AR annotations, virtual replicas). At present, remote experts need to carry out complex operations to transfer information to local users, but the fusion of virtual and real information makes the display of information in the MR collaborative interaction interface appear messy and redundant, and local users sometimes find it difficult to pay attention to the focus of information transferred by experts. Our research aims to simplify the operation of remote experts in MR remote collaborative assembly and to enhance the expression of visual cues that reflect experts’ attention, so as to promote the expression and communication of collaborative intention that user has and improve assembly efficiency. We developed a system (EaVAS) through a method that is based on the assembly semantic association model and the expert operation visual enhancement mechanism that integrates gesture, eye gaze, and spatial visual cues. EaVAS can give experts great freedom of operation in MR remote collaborative assembly, so that experts can strengthen the visual expression of the information they want to convey to local users. EaVAS was tested for the first time in an engine physical assembly task. The experimental results show that the EaVAS has better time performance, cognitive performance, and user experience than that of the traditional MR remote collaborative assembly method (3DGAM). Our research results have certain guiding significance for the research of user cognition in MR remote collaborative assembly, which expands the application of MR technology in collaborative assembly tasks.

show abstract

Section: Introductionmentioning

confidence: 99%

Can you notice my attention? A novel information vision enhancement method in MR remote collaborative assembly

Yan

Bai

et al. 2023

Int J Adv Manuf Technol

View full text Add to dashboard Cite

show abstract

View recommendation for multi-camera demonstration-based training

Biswas

Kruijff

Veas

2023

Multimed Tools Appl

View full text Add to dashboard Cite

While humans can effortlessly pick a view from multiple streams, automatically choosing the best view is a challenge. Choosing the best view from multi-camera streams poses a problem regarding which objective metrics should be considered. Existing works on view selection lack consensus about which metrics should be considered to select the best view. The literature on view selection describes diverse possible metrics. And strategies such as information-theoretic, instructional design, or aesthetics-motivated fail to incorporate all approaches. In this work, we postulate a strategy incorporating information-theoretic and instructional design-based objective metrics to select the best view from a set of views. Traditionally, information-theoretic measures have been used to find the goodness of a view, such as in 3D rendering. We adapted a similar measure known as the viewpoint entropy for real-world 2D images. Additionally, we incorporated similarity penalization to get a more accurate measure of the entropy of a view, which is one of the metrics for the best view selection. Since the choice of the best view is domain-dependent, we chose demonstration-based training scenarios as our use case. The limitation of our chosen scenarios is that they do not include collaborative training and solely feature a single trainer. To incorporate instructional design considerations, we included the trainer’s body pose, face, face when instructing, and hands visibility as metrics. To incorporate domain knowledge we included predetermined regions’ visibility as another metric. All of those metrics are taken into account to produce a parameterized view recommendation approach for demonstration-based training. An online study using recorded multi-camera video streams from a simulation environment was used to validate those metrics. Furthermore, the responses from the online study were used to optimize the view recommendation performance with a normalized discounted cumulative gain (NDCG) value of 0.912, which shows good performance with respect to matching user choices.

show abstract

Does mixed reality influence joint action? Impact of the mixed reality setup on users’ behavior and spatial interaction

Brument,

De Pace,

Podkosova

2024

J Multimodal User Interfaces

View full text Add to dashboard Cite

Understanding how people effectively perform actions together is fundamental when designing Collaborative Mixed Reality (CMR) applications. While most of the studies on CMR mostly considered either how users are immersed in the CMR (e.g., in virtual or augmented reality) or how the physical workspace is shared by users (i.e., distributed or collocated), little is known about how their combination could influence user’s interaction in CMR. In this paper, we present a user study (n = 46, 23 pairs) that investigates the effect of the mixed reality setup on the user’s immersion and spatial interaction during a joint-action task. Groups of two participants had to perform two types of joint actions while carrying a virtual rope to maintain a certain distance: (1) Gate, where participants had to pass through a virtual aperture together, and (2) Fruit, where participants had to use a rope to slice a virtual fruit moving in the CMR. Users were either in a distributed or collocated setup and either immersed in virtual or augmented reality. Our results showed that the immersion type and location setup altered users’ proxemics as well as the users’ subjective experience. In particular, we noticed better task performance when users were in augmented reality and more considerable distances between players while interacting in a distributed setup. These results contribute to the understanding of joint action in CMR and are discussed to improve the design of CMR applications.

show abstract

Spatial Perception Enhancement in Assembly Training Using Augmented Volumetric Playback

Cited by 9 publications

References 40 publications

Can you notice my attention? A novel information vision enhancement method in MR remote collaborative assembly

Can you notice my attention? A novel information vision enhancement method in MR remote collaborative assembly

View recommendation for multi-camera demonstration-based training

Does mixed reality influence joint action? Impact of the mixed reality setup on users’ behavior and spatial interaction

Contact Info

Product

Resources

About