Synthesising 2D Video from 3D Motion Data for Machine Learning Applications

Abstract

To increase the utility of legacy, gold-standard, three-dimensional (3D) motion capture datasets for computer vision-based machine learning applications, this study proposed and validated a method to synthesise two-dimensional (2D) video image frames from historic 3D motion data. We applied the video-based human pose estimation model OpenPose to real (in situ) and synthesised 2D videos and compared anatomical landmark keypoint outputs, with trivial observed differences (2.11–3.49 mm). We further demonstrated the utility of the method in a downstream machine learning use-case in which we trained and then tested the validity of an artificial neural network (ANN) to estimate ground reaction forces (GRFs) using synthesised and real 2D videos. Training an ANN to estimate GRFs using eight OpenPose keypoints derived from synthesised 2D videos resulted in accurate waveform GRF estimations (r > 0.9; nRMSE < 14%). When compared with using the smaller number of real videos only, accuracy was improved by adding the synthetic views and enlarging the dataset. The results highlight the utility of the developed approach to enlarge small 2D video datasets, or to create 2D video images to accompany 3D motion capture datasets to make them accessible for machine learning applications.

Estimating Ground Reaction Forces from Two-Dimensional Pose Data: A Biomechanics-Based Comparison of AlphaPose, BlazePose, and OpenPose

“…In the current study, we achieved a slightly higher accuracy than the LSTM neural network with just a single camera view (medio-lateral GRF , anterior–posterior GRF , and vertical GRF ) [ 17 ]. In another study [ 18 ], we synthesised five additional camera views surrounding the force plate and analysed the estimation accuracy for GRF in sidestepping. We achieved high estimation accuracy for all three force components (medio-lateral GRF , anterior–posterior GRF , and vertical GRF ).…”

“…A clear statement of the limitations of a machine learning model is of distinct importance given that they can be used to infer an athlete’s training load from broadcast footage with non-valid estimations leading to potentially harmful downstream decision making [ 40 ]. Furthermore, the use of remotely obtained video footage to estimate personal information such as load imposes a risk to the privacy and autonomy of an athlete [ 18 , 41 ]. In a clinical context, further validation is necessary with a population showing pathological pose and movement.…”

“…Portable technologies, such as pressure insoles, can be used in the field, but these technologies are limited to determine the vertical GRF component and have been reported to interfere with an athlete’s free movement [ 9 ]. To address these limitations, previous sports biomechanics researchers have trained machine learning models that seek to estimate GRF from motion kinematics collected using 3D retro-reflective optical motion capture [ 10 , 11 , 12 , 13 ], inertial sensors [ 8 , 14 , 15 , 16 ], and 2D video based pose estimation keypoints, themselves derived from other computer vision-based machine learning models [ 17 , 18 ]. Since traditional 3D optical motion capture is restricted to a laboratory environment and body-affixed inertial sensors pose a safety risk during competition and may also inhibit athlete motion, the focus of this study was to investigate the utility of 2D pose estimation algorithms for use in biomechanics.…”

Examination of 2D frontal and sagittal markerless motion capture: Implications for 2D and 3D markerless applications

“…A potentially more promising solution is to create realistic synthetic images 33,36,37 using virtual environments and body models from which 2D labelled images can be generated 33,38 . As a case study, Mundt, et al 36 used marker-based data to generate synthetic images of virtual human models performing a sidestepping task in three participants. While no formal analysis was performed, they did demonstrate similar markerless results between real and synthetic images, although this was only performed for the lower limbs with a blank grey background.…”

“…A potentially more promising solution is to create realistic synthetic images 33,36,37 using virtual environments and body models from which 2D labelled images can be generated 33,38 . As a case study, Mundt, et al 36 used marker-based data to generate synthetic images of virtual human models performing a sidestepping task in three participants.…”