Filter Guided Manifold Optimization in the Autoencoder Latent Space

Lannan, Nate; Fan, Guoliang

doi:10.1109/cvprw.2019.00125

Cited by 9 publications

(12 citation statements)

References 9 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The results of simulated data and real-world data are shown in Table 1 and 2. It is worth noting that we have compared the results of our methods with the results of the autoencoder-based approach [11], [12] and the Kalman filter-assisted autoencoder approach in [13]. The proposed TKF-DE outperforms all of them on the two datasets in terms of joint positions.…”

Section: Resultsmentioning

confidence: 99%

See 1 more Smart Citation

Human Motion Enhancement via Tobit Particle Filtering and Differential Evolution

Zhou

Lannan

Fan

2021

2021 IEEE International Conference on Image Processing (ICIP)

Self Cite

View full text Add to dashboard Cite

This paper proposes a novel approach to improve the quality of human motion data captured by a depth sensor. Depthbased motion capture (D-Mocap) data often suffer significant errors due to noise, self-occlusion, interference, and other algorithmic limitations. We aim to improve 3D joint trajectories to be more kinematically admissible and anthropometrically consistent. The Tobit model is incorporated with a particle filter (TPF) to handle censored measurements. We also embed the DE algorithm in the TPF, which allows particles to be re-distributed and re-weighted according to bone length consistency before re-sampling. This integration leads to a new TPF-DE algorithm that harmoniously takes advantage of kinematic and anthropometric constraints. We compare our methods with several nonlinear Kalman filters and deep learning-based methods to demonstrate the efficacy of TPF-DE on both simulated and real-world D-Mocap data.

show abstract

Section: Resultsmentioning

confidence: 99%

“…A combination of the KF and differential evolution (DE) was developed in [7], [23] with an attempt to smooth joint trajectories along with bone length preservation. In addition, in [12], [13] traditional filtering is integrated into a deep learning architecture as a target to improve the D-Mocap data.…”

Section: Introductionmentioning

confidence: 99%

Human Motion Enhancement via Tobit Particle Filtering and Differential Evolution

Zhou

Lannan

Fan

2021

2021 IEEE International Conference on Image Processing (ICIP)

Self Cite

View full text Add to dashboard Cite

show abstract

“…This is because the autoencoder is trained on a rich dataset of varied Mocap data to create a manifold of human motion, but this manifold does not take into consideration the kinematics inherent amongst the sequential data. Li et al use the addition of an LSTM to learn the natural kinematic dependencies of the original motion, whereas we have chosen to use the TKF to create a set of target motions that the data can be optimized toward in the latent space while adhering to the manifold learned by the convolutional autoencoder [29], [30]. Our TKF-assisted autoencoder builds upon the state of the art by combining the robust manifold obtained through deep learning and the preserved kinematics of human motion through recursive filtering.…”

Section: Deep Learning Methodsmentioning

confidence: 99%

Human Motion Enhancement via Tobit Kalman Filter-Assisted Autoencoder

2022

Self Cite

View full text Add to dashboard Cite

We present a novel approach to enhance the quality of human motion data collected by lowcost depth sensors, namely D-Mocap, which suffers from low accuracy and poor stability due to occlusion, interference, and algorithmic limitations. Our approach takes advantage of a large set of high-quality and diverse Mocap data by learning a general motion manifold via the convolutional autoencoder. In addition, the Tobit Kalman filter (TKF) is used to capture the kinematics of each body joint and handle censored measurement distribution. The TKF is incorporated with the autoencoder via latent space optimization, maintaining adherence to the motion manifold while preserving the kinematic nature of the original motion data. Furthermore, due to the lack of an open source benchmark dataset for this research, we have developed an extension of the Berkeley Multimodal Human Action Database (MHAD) by generating D-Mocap data from RGB-D images. The newly extended MHAD dataset is skeleton-matched and time-synced to the corresponding Mocap data and is publicly available. Along with simulated D-Mocap data generated from the CMU Mocap dataset and our self-collected D-Mocap dataset, the proposed algorithm is thoroughly evaluated and compared with different settings. Experimental results show that our approach can improve the accuracy of joint positions and angles as well as skeletal bone lengths by over 50%.INDEX TERMS Autoencoder, human motion manifold, depth sensors, motion capture, Tobit Kalman filter.

show abstract

“…A constraint KF method was developed in [18] which combine DE and the KF to refine the bone-lengths fluctuations with an attempt to improve the smoothness of joint trajectories. In addition, in [20], traditional filtering is used to create a target for optimization over the latent space of a convolutional autoencoder. The authors use the KF and deep learning to recover corrupted Mocap or low quality D-Mocap data.…”

Section: Eai Endorsed Transactions On Bioengineering and Bioinformaticsmentioning

confidence: 99%

“…Some examples of learning methods in D-Mocap enhancement are, Differential Evolution (DE) used in [17] and Genetic Algorithms (GA) used in [18], and in the case of deep learning-based methods, the improvement of D-Mocap data through a learned motion manifold in [19]. There have been a few attempt to integrate kinematic and anthropometric constraints with the purpose of improving joint trajectories over time and the skeleton structure in space [17,18,20].…”

Section: Introductionmentioning

confidence: 99%

Human Motion Enhancement via Joint Optimization of Kinematic and Anthropometric Constraints

Zhou¹,

Lannan²,

Fan³

et al. 2021

EAI Endorsed Transactions on Bioengineering and Bioinformatics

Self Cite

View full text Add to dashboard Cite

INTRODUCTION:Recently, low cost RGB-D depth sensors have emerged as a promising alternative for motion capture for clinical gait assessment. However, depth sensor based Mocap (D-Mocap) suffers from low accuracy and poor stability for 3D joint estimation due to noise, self-occlusion, interference, and other technical limitations, which prevents it from being widely used in health-related applications. OBJECTIVES: The primary objective of this study is to integrate non-linear Kalman filters (KFs) and evolutionary algorithms to enhance the quality of D-Mocap data by jointly considering kinematic or anthropometric constraints at the joint-level and skeleton-level, respectively. METHODS: We propose a hybrid approach to synergistically integrate the Tobit Kalman filter (TKF) and the Differential Evolutionary (DE) algorithm for human motion enhancement that is referred to as TKF-DE. Specifically, the joint-level TKF provides the predictive distribution of each joint that is kinematically admissible in time and probabilistically amenable in space for skeleton-level DE optimization in terms of all bone lengths. Two predictive distributions of the TKF, i.e., the Gaussian and uniform, are tested and compared in terms of their effectiveness in generating the initial DE population. RESULTS: Two sets of motion capture data are used to validate the proposed TKF-DE methods, one simulated and one real-world. The first dataset is from the Carnegie Mellon University Database (CMU) which contains a multitude of various motions and is simplified to a 21-joint skeleton and corrupted with additive white Gaussian noise (AWGN). The second dataset was collected at two labs at Oklahoma State University (OSU). The Orbbec depth sensor and the Nuitrack SDK were used for D-Mocap data acquisition along with an optical Mocap system that was time-synchronized and skeleton-matched with the D-Mocap system as a reference for evaluation. The results confirm that the proposed TKF-DE algorithms significantly outperform other nonlinear KFs, including the extended KF (EKF), the unscented KF (UKF), and the TKF alone, with improved accuracy and stability of the estimation of joint positions, bone lengths, and joint angles. It is also shown that the Gaussian-based predictive distribution is better than the uniform one, further validating the efficacy and synergy of the two key components in the TKF-DE algorithm. CONCLUSION: Our research synergistically integrates the TKF and DE algorithms in one framework referred to as the TKF-DE, that takes advantage of kinematic and anthropometric constraints for human motion enhancement. The experimental results on two D-Mocap datasets show that the proposed TKF-DE method significantly improves the quality of D-Mocap data in terms of joint positions, bone lengths, and joint angles. This study takes one step closer to bringing the D-Mocap technology to possible health-related applications.

show abstract

Filter Guided Manifold Optimization in the Autoencoder Latent Space

Cited by 9 publications

References 9 publications

Human Motion Enhancement via Tobit Particle Filtering and Differential Evolution

Human Motion Enhancement via Tobit Particle Filtering and Differential Evolution

Human Motion Enhancement via Tobit Kalman Filter-Assisted Autoencoder

Human Motion Enhancement via Joint Optimization of Kinematic and Anthropometric Constraints

Contact Info

Product

Resources

About