LiDAR-aid Inertial Poser: Large-scale Human Motion Capture by Sparse Inertial and LiDAR Sensors

Ren, Yiming; Zhao, Chengfeng; He, Yannan; Cong, Peishan; Liang, Han; Yu, Jingyi; Xu, Lu; Ma, Yuexin

doi:10.1109/tvcg.2023.3247088

Cited by 21 publications

(7 citation statements)

References 73 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…However, camerabased Mocap systems face challenges such as occlusions in certain camera views [23]. To overcome these problems, some researchers have designed hybrid systems that combine different Mocap technologies [33,34] to improve accuracy and reduce camera occlusions. As previously mentioned, autonomous control of drones requires high-frequency position and orientation information, so the structure should be as simple and efficient as possible.…”

Section: Motion Capture Methodsmentioning

confidence: 99%

Easy Rocap: A Low-Cost and Easy-to-Use Motion Capture System for Drones

Wang,

Chen,

et al. 2024

Drones

View full text Add to dashboard Cite

Fast and accurate pose estimation is essential for the local motion control of robots such as drones. At present, camera-based motion capture (Mocap) systems are mostly used by robots. However, this kind of Mocap system is easily affected by light noise and camera occlusion, and the cost of common commercial Mocap systems is high. To address these challenges, we propose Easy Rocap, a low-cost, open-source robot motion capture system, which can quickly and robustly capture the accurate position and orientation of the robot. Firstly, based on training a real-time object detector, an object-filtering algorithm using class and confidence is designed to eliminate false detections. Secondly, multiple-object tracking (MOT) is applied to maintain the continuity of the trajectories, and the epipolar constraint is applied to multi-view correspondences. Finally, the calibrated multi-view cameras are used to calculate the 3D coordinates of the markers and effectively estimate the 3D pose of the target robot. Our system takes in real-time multi-camera data streams, making it easy to integrate into the robot system. In the simulation scenario experiment, the average position estimation error of the method is less than 0.008 m, and the average orientation error is less than 0.65 degrees. In the real scenario experiment, we compared the localization results of our method with the advanced LiDAR-Inertial Simultaneous Localization and Mapping (SLAM) algorithm. According to the experimental results, SLAM generates drifts during turns, while our method can overcome the drifts and accumulated errors of SLAM, making the trajectory more stable and accurate. In addition, the pose estimation speed of our system can reach 30 Hz.

show abstract

Section: Motion Capture Methodsmentioning

confidence: 99%

Easy Rocap: A Low-Cost and Easy-to-Use Motion Capture System for Drones

Wang,

Chen,

et al. 2024

Drones

View full text Add to dashboard Cite

show abstract

“…LidarCap (Li et al 2022a) achieves remarkable results in 3D human motion capture within a range of 30 meters, relying solely on individual human point clouds obtained from a single monocular LiDAR sensor, demonstrating its outstanding potential. Multimodal fusion methods (Ren et al 2022;Fürst et al 2020;Kim et al 2019) improve algorithm robustness and accuracy by combining LiDAR data with data from other modalities. Nevertheless, previous methods overlook the fact that LiDAR sensors capture data not only from humans but also from the environment.…”

Section: Bn 3sn Humanmentioning

confidence: 99%

Neighborhood-Enhanced 3D Human Pose Estimation with Monocular LiDAR in Long-Range Outdoor Scenes

Zhang,

Mao,

et al. 2024

AAAI

View full text Add to dashboard Cite

3D human pose estimation (3HPE) in large-scale outdoor scenes using commercial LiDAR has attracted significant attention due to its potential for real-life applications. However, existing LiDAR-based methods for 3HPE primarily rely on recovering 3D human poses from individual point clouds, and the coherence cues present in the neighborhood are not sufficiently harnessed. In this work, we explore spatial and contexture coherence cues contained in the neighborhood that lead to great performance improvements in 3HPE. Specifically, firstly, we deeply investigate the 3D neighbor in the background (3BN) which serves as a spatial coherence cue for inferring reliable motion since it provides physical laws to limit motion targets. Secondly, we introduce a novel 3D scanning neighbor (3SN) generated during the data collection and 3SN implies structural edge coherence cues. We use 3SN to overcome the degradation of performance and data quality caused by the sparsity-varying properties of LiDAR point clouds. In order to effectively model the complementation between these distinct cues and build consistent temporal relationships across human motions, we propose a new transformer-based module called the CoherenceFuse module. Extensive experiments were conducted on publicly available datasets, namely LidarHuman26M, CIMI4D, SLOPER4D and Waymo Open Dataset v2.0, showcase the superiority and effectiveness of our proposed method. In particular, when compared with LidarCap on the LidarHuman26M dataset, our method demonstrates a reduction of 7.08mm in the average MPJPE metric, along with a decrease of 16.55mm in the MPJPE metric for distances exceeding 25 meters. The code and models are available at https://github.com/jingyi-zhang/Neighborhood-enhanced-LidarCap.

show abstract

“…LiDARs offer precise depth measurements, making them well‐suited for large‐scale environmental measurements for autonomous driving scenes [LVC*19, SGJ*20]. Recent studies have delved into utilizing LiDAR for capturing detailed 3D human poses [RZH*23]. Moreover, sensor fusion approaches combining LiDAR and cameras have been proposed [CXR*22, RZH*23] to leverage the complementary strengths of these sensors.…”

Section: Related Workmentioning

confidence: 99%

“…Recent studies have delved into utilizing LiDAR for capturing detailed 3D human poses [RZH*23]. Moreover, sensor fusion approaches combining LiDAR and cameras have been proposed [CXR*22, RZH*23] to leverage the complementary strengths of these sensors. However, these methods primarily focus on scene‐level tasks like human detection and segmentation, rather than capturing skeletal motions with precise global translation.…”

Section: Related Workmentioning

confidence: 99%

MOVIN: Real‐time Motion Capture using a Single LiDAR

Jang,

Yang,

Jang

et al. 2023

Computer Graphics Forum

View full text Add to dashboard Cite

Recent advancements in technology have brought forth new forms of interactive applications, such as the social metaverse, where end users interact with each other through their virtual avatars. In such applications, precise full‐body tracking is essential for an immersive experience and a sense of embodiment with the virtual avatar. However, current motion capture systems are not easily accessible to end users due to their high cost, the requirement for special skills to operate them, or the discomfort associated with wearable devices. In this paper, we present MOVIN, the data‐driven generative method for real‐time motion capture with global tracking, using a single LiDAR sensor. Our autoregressive conditional variational autoencoder (CVAE) model learns the distribution of pose variations conditioned on the given 3D point cloud from LiDAR. As a central factor for high‐accuracy motion capture, we propose a novel feature encoder to learn the correlation between the historical 3D point cloud data and global, local pose features, resulting in effective learning of the pose prior. Global pose features include root translation, rotation, and foot contacts, while local features comprise joint positions and rotations. Subsequently, a pose generator takes into account the sampled latent variable along with the features from the previous frame to generate a plausible current pose. Our framework accurately predicts the performer's 3D global information and local joint details while effectively considering temporally coherent movements across frames. We demonstrate the effectiveness of our architecture through quantitative and qualitative evaluations, comparing it against state‐of‐the‐art methods. Additionally, we implement a real‐time application to showcase our method in real‐world scenarios. MOVIN dataset is available at https://movin3d.github.io/movin_pg2023/https://movin3d.github.io/movin_pg2023/">https://movin3d.github.io/movin_pg2023/.

show abstract

LiDAR-aid Inertial Poser: Large-scale Human Motion Capture by Sparse Inertial and LiDAR Sensors

Cited by 21 publications

References 73 publications

Easy Rocap: A Low-Cost and Easy-to-Use Motion Capture System for Drones

Easy Rocap: A Low-Cost and Easy-to-Use Motion Capture System for Drones

Neighborhood-Enhanced 3D Human Pose Estimation with Monocular LiDAR in Long-Range Outdoor Scenes

MOVIN: Real‐time Motion Capture using a Single LiDAR

Contact Info

Product

Resources

About