ControllerPose: Inside-Out Body Capture with VR Controller Cameras

Ahuja, Karan; Shen, Vivian; Fang, Cathy Mengying; Riopelle, Nathan; Kong, Andy; Harrison, Chris

doi:10.1145/3491102.3502105

Cited by 26 publications

(6 citation statements)

References 45 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In contrast to external body pose capture systems that include high-end commercial marker-based camera systems [42], depth and RGB [10,16,39] cameras as well as non-optical systems [26,67,68], body-worn systems enable tracking in mobile settings, and do not require the user to remain within a constraint area. To receive a wider view of the user's body for tracking, cameras with wide angle-lenses were attached to various parts of the body including the wrist [62], the feet [7], and the chest [25,27] or alternatively integrated into controllers [3] or using a suspension on the headset [1,46] further away from the user's body. However, these solutions tend to suffer from occlusion and are often obtrusive to wear.…”

Section: Body Capture Using Worn Sensorsmentioning

confidence: 99%

HOOV: Hand Out-Of-View Tracking for Proprioceptive Interaction using Inertial Sensing

Streli

Armani

Cheng

et al. 2023

Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems

View full text Add to dashboard Cite

HOOV is a wireless sensing method that complements existing virtual and augmented reality headsets to support hand tracking outside the field of view of the headset's cameras. In this example app of our system, a participant is building a structure composed of blocks with different sizes. When the user places an object onto the structure, his hand is tracked by the headset's cameras (left). As the user reaches for a block just to his right (right), his hand leaves the field of view of the cameras. Now, HOOV continuously estimates the 6D position and orientation of the wrist using the 6-axis inertial measuring unit attached to the wrist during the time the hand stays outside the field of view.

show abstract

Section: Body Capture Using Worn Sensorsmentioning

confidence: 99%

HOOV: Hand Out-Of-View Tracking for Proprioceptive Interaction using Inertial Sensing

Streli

Armani

Cheng

et al. 2023

Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems

View full text Add to dashboard Cite

show abstract

“…In the context of metaverse VR applications, apart from many studies on its enabling technologies [18,27,38,48] such as AR/VR [1,24,26,29,30,34,42], motion sensing [3], and platform governance [9,10,19], their architectural problems and scalability issues to be solved by metaverse developers (e.g., bandwidth consumptions with increasing number of users, CPU/GPU utilization of terminal device, and rendering cost of user avatars) have been studied in [13,14]. The authors of [14] have also pointed out that one critical scalability issue that could be solved by changing the operational architecture of metaverses is that the consumptions of user bandwidth and CPU/GPU resources currently depend on the number of active users, thus, can introduce high overhead on the VR headset -one potential solution is remote rendering, i.e., rendering application graphics on the cloud server instead of on user VR devices.…”

Section: Related Workmentioning

confidence: 99%

MetaVRadar: Measuring Metaverse Virtual Reality Network Activity

Lyu,

Tripathi,

Sivaraman

2023

Proc. ACM Meas. Anal. Comput. Syst.

View full text Add to dashboard Cite

The "metaverse", wherein users can enter virtual worlds to work, study, play, shop, socialize, and entertain, is fast becoming a reality, attracting billions of dollars in investment from companies such as Meta, Microsoft, and Clipo Labs. Further, virtual reality (VR) headsets from entities like Oculus, HTC, and Microsoft are rapidly maturing to provide fully immersive experiences to metaverse users. However, little is known about the network dynamics of metaverse VR applications in terms of service domains, flow counts, traffic rates and volumes, content location and latency, etc., which are needed to make telecommunications network infrastructure "metaverse ready" to support superlative user experience in the coming future. This paper is an empirical measurement study of metaverse VR network behavior aimed at helping telecommunications network operators better provision and manage the network to ensure good user experience. Using illustrative hour-long network traces of metaverse sessions on the Oculus VR headset, we first develop a categorization of user activity into distinct states ranging from login home to streetwalking and event attendance to asset trading, and undertake a detailed analysis of network traffic per state, identifying unique service domains, protocols, flow profiles, and volumetric patterns, thereby highlighting the vastly more complex nature of a metaverse session compared to streaming video or gaming. Armed with the network behavioral profiles, our second contribution develops a real-time methodMetaVRadar to detect metaverse session and classify the user activity state leveraging formalized flow signatures and volumetric attributes. Our third contribution practically implementsMetaVRadar, evaluates its accuracy in our lab environment, and demonstrates its usability in a large university network so operators can better monitor and plan resources to support requisite metaverse user experience.

show abstract

“…Various user input data from consumer‐grade devices can be used to synthesize full‐body animation for characters in real time. For example, some studies used optical data from egocentric cameras mounted in a baseball cap [XCZ*19], a HMD [TAP*20; YCQ*22], controllers [ASF*22], or glasses [ZWMF21] to estimate the body pose. Egocentric cameras suffer from extreme perspective distortion and self‐occlusion that lead to inadequate tracking information for the lower body.…”

Section: Related Workmentioning

confidence: 99%

Combining Motion Matching and Orientation Prediction to Animate Avatars for Consumer‐Grade VR Devices

Luis

Yun

Andújar

et al. 2022

Computer Graphics Forum

View full text Add to dashboard Cite

The animation of user avatars plays a crucial role in conveying their pose, gestures, and relative distances to virtual objects or other users. Self‐avatar animation in immersive VR helps improve the user experience and provides a Sense of Embodiment. However, consumer‐grade VR devices typically include at most three trackers, one at the Head Mounted Display (HMD), and two at the handheld VR controllers. Since the problem of reconstructing the user pose from such sparse data is ill‐defined, especially for the lower body, the approach adopted by most VR games consists of assuming the body orientation matches that of the HMD, and applying animation blending and time‐warping from a reduced set of animations. Unfortunately, this approach produces noticeable mismatches between user and avatar movements. In this work we present a new approach to animate user avatars that is suitable for current mainstream VR devices. First, we use a neural network to estimate the user's body orientation based on the tracking information from the HMD and the hand controllers. Then we use this orientation together with the velocity and rotation of the HMD to build a feature vector that feeds a Motion Matching algorithm. We built a MoCap database with animations of VR users wearing a HMD and used it to test our approach on both self‐avatars and other users' avatars. Our results show that our system can provide a large variety of lower body animations while correctly matching the user orientation, which in turn allows us to represent not only forward movements but also stepping in any direction.

show abstract

ControllerPose: Inside-Out Body Capture with VR Controller Cameras

Cited by 26 publications

References 45 publications

HOOV: Hand Out-Of-View Tracking for Proprioceptive Interaction using Inertial Sensing

HOOV: Hand Out-Of-View Tracking for Proprioceptive Interaction using Inertial Sensing

MetaVRadar: Measuring Metaverse Virtual Reality Network Activity

Combining Motion Matching and Orientation Prediction to Animate Avatars for Consumer‐Grade VR Devices

Contact Info

Product

Resources

About