Online Learning for the Hyoid Bone Tracking During Swallowing With Neck Movement Adjustment Using Semantic Segmentation

Abstract

Swallowing difficulty is a major health concern of the elderly population. The gold standard examination to assess swallowing function is videofluoroscopic swallowing study (VFSS). Hyoid kinematic parameters extracted from VFSS images can be quantitative indicators of swallowing difficulty. In previous studies, its tracking failures are still not resolved when passing through the mandible. Furthermore, it is difficult to be applied in kinematic analysis because the hyoid trajectories can be susceptible to irrelevant neck movements during swallowing. The aim of this study is to develop a robust algorithm for obtaining high-accuracy trajectories of the hyoid bone during swallowing with adjustment of the neck movements. We propose a CNN-based hyoid tracking algorithm which consists of single-domain networks for hyoid tracking and an attention U-Net with conditional random fields for semantic segmentation of the hyoid bone and the cervical vertebrae. The results show that the proposed method can track the hyoid bone robustly compared to the previous methods as measured by a success plot of one-pass evaluation. In addition, the proposed semantic segmentation method achieved the highest dice coefficient for the hyoid bone and the cervical vertebrae. Finally, the obtained hyoid trajectories were evaluated by a root mean squared error, relative error of range of motion, and Pearson's correlation analysis. The proposed algorithm can provide ability to automatically analyze the hyoid motions during swallowing in clinical practice and will potentially enable physician's decision making on diagnostic and therapeutic modalities based on quantitative swallowing assessments.

Automatic Hyoid Bone Tracking in Real-Time Ultrasound Swallowing Videos Using Deep Learning Based and Correlation Filter Based Trackers

“…SiamFC trained with USV gave an RMSE of 3.85 pixels ± 1.06 pixels (1.25 mm ± 0.34 mm) and an AE of 3.28 pixels ± 1.10 pixels (1.07 mm ± 0.36 mm). This result appears to outperform a reported RMSE of 3.2 mm ± 0.4 mm in a previous study using deep learning trackers on VFSS [ 23 ].…”

“…Lopes et al (2019) used You Only Look Once version 3 (YOLOv3)to locate the hyoid bone in the ultrasound imaging [ 21 ], which gives some insights on automatically labeling the hyoid location in one single ultrasound image, yet they did not test tracking of hyoid bone locations in subsequent frames in ultrasound videos. Detection and segmentation-related deep learning methods have been applied to track the locations of the hyoid bone in VFSS [ 22 , 23 ]. However, tracking-related deep learning methods have not been applied to ultrasound videos.…”

Hyoid Bone Tracking in a Videofluoroscopic Swallowing Study Using a Deep-Learning-Based Segmentation Network

“…As deep learning technology has been developed and used for fast and efficient analysis of medical images acquired by techniques such as X-ray, computed tomography (CT), and magnetic resonance imaging (MRI) [ 18 , 19 , 20 ], recent studies have tried to apply deep learning to automate VFSS analysis [ 21 , 22 , 23 , 24 , 25 , 26 , 27 ]. However, we found only two studies that proposed deep learning models to detect the hyoid bone or track its movement in VFSS images [ 21 , 27 ].…”

“…As deep learning technology has been developed and used for fast and efficient analysis of medical images acquired by techniques such as X-ray, computed tomography (CT), and magnetic resonance imaging (MRI) [ 18 , 19 , 20 ], recent studies have tried to apply deep learning to automate VFSS analysis [ 21 , 22 , 23 , 24 , 25 , 26 , 27 ]. However, we found only two studies that proposed deep learning models to detect the hyoid bone or track its movement in VFSS images [ 21 , 27 ]. Zhang et al proposed the single shot multibox detector (SSD) model that can detect the hyoid bone fully automatically, but it showed less than perfect accuracy (mAP of the SSD-500 model = 89.14%), and tracking the whole movement of the hyoid bone was not attempted [ 27 ].…”

“…Zhang et al proposed the single shot multibox detector (SSD) model that can detect the hyoid bone fully automatically, but it showed less than perfect accuracy (mAP of the SSD-500 model = 89.14%), and tracking the whole movement of the hyoid bone was not attempted [ 27 ]. Lee et al proposed a convolutional-neural-network (CNN)-based online learning algorithm, which can track hyoid motion and predict it even when it passes through the mandible and is blurred, but it still requires manual demarcation of a hyoid bone of appropriate size by humans in the first frame [ 21 ].…”