Beyond the Edge: Markerless Pose Estimation of Speech Articulators from Ultrasound and Camera Images Using DeepLabCut

Abstract

Automatic feature extraction from images of speech articulators is currently achieved by detecting edges. Here, we investigate the use of pose estimation deep neural nets with transfer learning to perform markerless estimation of speech articulator keypoints using only a few hundred hand-labelled images as training input. Midsagittal ultrasound images of the tongue, jaw, and hyoid and camera images of the lips were hand-labelled with keypoints, trained using DeepLabCut and evaluated on unseen speakers and systems. Tongue surface contours interpolated from estimated and hand-labelled keypoints produced an average mean sum of distances (MSD) of 0.93, s.d. 0.46 mm, compared with 0.96, s.d. 0.39 mm, for two human labellers, and 2.3, s.d. 1.5 mm, for the best performing edge detection algorithm. A pilot set of simultaneous electromagnetic articulography (EMA) and ultrasound recordings demonstrated partial correlation among three physical sensor positions and the corresponding estimated keypoints and requires further investigation. The accuracy of the estimating lip aperture from a camera video was high, with a mean MSD of 0.70, s.d. 0.56, mm compared with 0.57, s.d. 0.48 mm for two human labellers. DeepLabCut was found to be a fast, accurate and fully automatic method of providing unique kinematic data for tongue, hyoid, jaw, and lips.

Articulatory and acoustic dynamics of fronted back vowels in American English

“…Still ultrasound images were aligned to the audio recording using pulses generated by the Articulate Instruments PStretch unit, recorded in AAA alongside the speech signal. Tongue splines were automatically fit with DeepLabCut (Mathis et al, 2018;Nath et al, 2019) using the MobileNet1.0-based neural network implemented in AAA (Wrench and Balch-Tomes, 2022). Tongue coordinates were rotated to a common horizontal plane (Lawson et al, 2019;Scobbie et al, 2011) by visually estimating the orientation of the ultrasound probe and camera from the side-view lip video data.…”

A systematic review of the applications of markerless motion capture (MMC) technology for clinical measurement in rehabilitation

“…Human pose estimation technology such as Convolutional Pose Machines (CPM) and convolution neural network (CNN) based methods which allow extraction of human movement information directly from video clips have been repeatedly tested by researchers [ 85 , 86 ] while human pose estimation application on analyzing movement in the disease populations were reported to be useful by the studies in our review [ 14 , 16 – 25 , 27 , 29 , 32 – 38 , 41 , 44 , 50 – 55 , 57 , 66 , 71 – 74 ]. Given that such trajectory extraction method is in rapid evaluation and is becoming more mature for promising identification of posture [ 87 – 89 ], using hand-held camera or smartphone as the MMC system would be especially beneficial for understanding the motor performance of individuals in their daily living tasks, hence providing valuable information on levels of impairment and on the constraints that patients might encounter in their activities of daily living in their real-life environment. It is understandable that individuals, particularly young children and older people, might behave differently when they are placed for motion capturing in an unfamiliar laboratory or a simulated environment, thus risking the possibility that the motion analysis might not truly reflect the individuals’ actual movement patterns [ 90 ].…”

A comparison of point-tracking algorithms in ultrasound videos from the upper limb

“…While DLC has been used extensively to track animal and human features during movements, its ability to track features in US videos has been minimally explored. [ 46 ] used DLC to track the upper surface of the tongue and compared it to other US contour estimators, concluding that DLC requires significantly less training data to perform with the same level of accuracy. [ 47 ] used DLC to track the gastrocnemius muscle–tendon junction, observing the morphology of the lower leg longitudinally.…”

“…The manually labeled data of both Group 1 and Group 2 was restructured into the appropriate file types for DLC to use for training. The training data was augmented using the imgaug method ( https://github.com/aleju/imgaug ) and a 50-layer ResNet network was re-trained using this data for 500 k iterations, where error commonly plateaus [ 46 ] for ResNet50.…”