An Avatar Robot Overlaid with the 3D Human Model of a Remote Operator

We present a novel dataset for humanoid robot pose estimation from images, addressing the critical need for accurate pose estimation to enhance human–robot interaction in extended reality (XR) applications. Despite the importance of this task, large-scale pose datasets for diverse humanoid robots remain scarce. To overcome this limitation, we collected sparse pose datasets for commercially available humanoid robots and augmented them through various synthetic data generation techniques, including AI-assisted image synthesis, foreground removal, and 3D character simulations. Our dataset is the first to provide full-body pose annotations for a wide range of humanoid robots exhibiting diverse motions, including side and back movements, in real-world scenarios. Furthermore, we introduce a new benchmark method for real-time full-body 2D keypoint estimation from a single image. Extensive experiments demonstrate that our extended dataset-based pose estimation approach achieves over 33.9% improvement in accuracy compared to using only sparse datasets. Additionally, our method demonstrates the real-time capability of 42 frames per second (FPS) and maintains full-body pose estimation consistency in side and back motions across 11 differently shaped humanoid robots, utilizing approximately 350 training images per robot.

show abstract

Optimized Data Transmission and Signal Processing for Telepresence Suits in Multiverse Interactions

Volkov,

Muthanna,

Paramonov

et al. 2024

JSAN

View full text Add to dashboard Cite

With the rapid development of the metaverse, designing effective interfaces in virtual and augmented environments presents significant challenges. Additionally, keeping real-time sensory data flowing from users to their virtual avatars in a seamless and accurate manner is one of the biggest challenges in this domain. To this end, this article investigates a telepresence suit as an interface for interaction within the metaverse and its virtual avatars, aiming to address the complexities of signal generation, conversion, and transmission in real-time telepresence systems. We model a telepresence suit framework that systematically generates state data and transmits it to end-points, which can be either robotic avatars or virtual representations within a metaverse environment. Through a hand movement study, we successfully minimized the volume of transmitted information, reducing traffic by over 50%, which directly decreased channel load and packet delivery delay. For instance, as channel load decreases from 0.8 to 0.4, packet delivery delay is reduced by approximately half. This optimization not only enhances system responsiveness but also improves accuracy, particularly by reducing delays and errors in high-priority signal paths, enabling more precise and reliable telepresence interactions in metaverse settings.

show abstract

An Avatar Robot Overlaid with the 3D Human Model of a Remote Operator

Cited by 4 publications

References 17 publications

Does Robot Anthropomorphism Improve Performance and User Experience in Teleoperation?

Does Robot Anthropomorphism Improve Performance and User Experience in Teleoperation?

Diverse Humanoid Robot Pose Estimation from Images Using Only Sparse Datasets

Optimized Data Transmission and Signal Processing for Telepresence Suits in Multiverse Interactions

Contact Info

Product

Resources

About