Weakly Supervised Deep Learning for Tooth-Marked Tongue Recognition

Abstract

The recognition of tooth-marked tongues has important value for clinical diagnosis of traditional Chinese medicine. Tooth-marked tongue is often related to spleen deficiency, cold dampness, sputum, effusion, and blood stasis. The clinical manifestations of patients with tooth-marked tongue include loss of appetite, borborygmus, gastric distention, and loose stool. Traditional clinical tooth-marked tongue recognition is conducted subjectively based on the doctor’s visual observation, and its performance is affected by the doctor’s subjectivity, experience, and environmental lighting changes. In addition, the tooth marks typically have various shapes and colors on the tongue, which make it very challenging for doctors to identify tooth marks. The existing methods based on deep learning have made great progress for tooth-marked tongue recognition, but there are still shortcomings such as requiring a large amount of manual labeling of tooth marks, inability to detect and locate the tooth marks, and not conducive to clinical diagnosis and interpretation. In this study, we propose an end-to-end deep neural network for tooth-marked tongue recognition based on weakly supervised learning. Note that the deep neural network only requires image-level annotations of tooth-marked or non-tooth marked tongues. In this method, a deep neural network is trained to classify tooth-marked tongues with the image-level annotations. Then, a weakly supervised tooth-mark detection network (WSTDN) as an architecture variant of the pre-trained deep neural network is proposed for the tooth-marked region detection. Finally, the WSTDN is re-trained and fine-tuned using only the image-level annotations to simultaneously realize the classification of the tooth-marked tongue and the positioning of the tooth-marked region. Experimental results of clinical tongue images demonstrate the superiority of the proposed method compared with previously reported deep learning methods for tooth-marked tongue recognition. The proposed tooth-marked tongue recognition model may provide important syndrome diagnosis and efficacy evaluation methods, and contribute to the understanding of ethnopharmacological mechanisms.

Development of a tongue image-based machine learning tool for the diagnosis of gastric cancer: a prospective multicentre clinical cohort study

“…The theory of TCM suggests that changes detected using tongue images (the colour, size and shape of the tongue and the colour, thickness and moisture content of the tongue coating) can reflect the health state of the human body, which is especially closely related to gastric diseases. 4 , 5 , 6 Recent studies have shown that changes in tongue images and tongue coatings are closely related to the oral/tongue coating microbiome. 7 Additionally, many studies have confirmed that the oral/tongue coating microbial group has good diagnostic value for pancreatic cancer, 8 liver carcinoma, 9 colorectal cancer 10 and other tumours, as well as gastritis, 11 rheumatoid arthritis, 12 chronic hepatitis B 13 and other diseases.…”

A survey of artificial intelligence in tongue image for disease diagnosis and syndrome differentiation

“…However, the symptoms on the tongue are very difficult to automatically identify or quantify, which has become a core challenge in SD. In recent years, many studies in this area have paid much attention to using deep learning methods to automatically classify or identify tongue symptoms, like tooth-marked tongue, 11 , 15 , 103 – 105 , 110 , 111 tongue coating, 10 , 16 , 31 , 42 , 112 , 113 coating color, 66 , 113 , 114 tongue color, 22 , 113 , 115 cracked tongue, 9 , 110 , 111 , 116 , 117 sublingual vein, 10 and fungiform papillae on tongue. 118 …”

“…Similarly, they utilized this method to classify rotten greasy coating of the tongue 16 and cracked tongue 116 based on the deep features, which outperforms other state-of-the-art methods. Another tooth-marked classification model based on image-level annotation was proposed by Zhou et al, 105 and it could also locate the tooth-marked area with the weakly supervised method. When classifying coating features, Wang et al 31 proposed a GreasyCoatNet framework that could classify three-level greasy coating robustly, indicating the potential ability to quantify tongue greasy coating.…”

Development of attention-based robust deep learning model for tongue diagnosis by smartphone

“…Hou et al conducted research on the classification of tongue color based on CNN with a data set of 1500 tongue images, and the accuracy rate was 89% [20]. In addition, some scholars have conducted methodological studies for different subtasks of tongue diagnosis, such as tongue color, fissure, tongue shape, and tongue coating [21][22][23][24][25][26][27]. The above study achieved high accuracy of tongue diagnosis based on TIAI.…”