Multimodal Semantic Segmentation: Fusion of RGB and Depth Data in Convolutional Neural Networks

Koppanyi, Zoltan; Iwaszczuk, D.; Zha, Bing; Saul, Can Jozef; Toth, Charles K.; Yilmaz, Alper

doi:10.1016/b978-0-12-817358-9.00009-3

Cited by 7 publications

(2 citation statements)

References 12 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Current trends in dentistry support using AI technology combined with intraoral photographs to diagnose caries lesions. 32 Our study fortified that dental students can be trained with AI-supported applications to improve their caries detection diagnostic skills.…”

Section: Discussionmentioning

confidence: 89%

Dental student application of artificial intelligence technology in detecting proximal caries lesions

Ayan,

Bayraktar,

Çelik

et al. 2024

Journal of Dental Education

View full text Add to dashboard Cite

ObjectivesThis study aimed to investigate the caries diagnosis performances of dental students after training with an artificial intelligence (AI) application utilizing deep learning techniques, a type of artificial neural network.MethodsA total of 1200 bitewing radiographs were obtained from the institution's database and two specialist dentists labeled the caries lesions in the images. Randomly selected 1000 images were used for training purposes and the remaining 200 radiographs were used to evaluate the caries diagnostic performance of the AI. Then, a convolutional neural network, a deep learning algorithm commonly employed to analyze visual imagery problems, called “You Only Look Once,” was modified and trained to detect enamel and dentin caries lesions in the radiographs. Forty dental students were selected voluntarily and randomly divided into two groups. The pre‐test results of dental caries diagnosis performances of both groups were recorded. After 1 week, group 2 students were trained using an AI application. Then, the post‐test results of both groups were recorded. The labeling duration of the students was also measured and analyzed.ResultsWhen both groups’ pre‐test and post‐test results were evaluated, a statistically significant improvement was found for all parameters examined except precision score (p < 0.05). However, the trained group's accuracy, sensitivity, specificity, and F1 scores were significantly higher than the non‐trained group in terms of post‐test scores (p < 0.05). In group 2 (trained group), the post‐test labeling time was considerably increased (p < 0.05).ConclusionsThe students trained by AI showed promising results in detecting caries lesions. The use of AI can also contribute to the clinical education of dental students.

show abstract

Section: Discussionmentioning

confidence: 89%

Dental student application of artificial intelligence technology in detecting proximal caries lesions

Ayan,

Bayraktar,

Çelik

et al. 2024

Journal of Dental Education

View full text Add to dashboard Cite

show abstract

“…This depth data can be also utilized as normal-based representation, which tend to deliver better accuracy. 14 Another modification 15 of FuseNet, called FuseNet234 and FuseNet-lite, was applied to a pair of three-band synthetic images with overlapping channels. Up to six channels were used, subdivided into two triplets, 16 each of them processed by the SegNet workflow.…”

Section: Previous Workmentioning

confidence: 99%

Exploring fusion techniques in U-Net and DeepLab V3 architectures for multi-modal land cover classification

Qiu

Budde²,

Bulatov³

et al. 2022

Earth Resources and Environmental Remote Sensing/Gis Applications XIII

View full text Add to dashboard Cite

Many deep learning architectures exist for semantic segmentation. In this paper, their application to multi-modal remote sensing data is examined. Two well-known network architectures, U-Net and DeepLab V3+, developed originally for RGB image data, are modified to accept additional input channels, such as near infrared or depth information. In both networks, ResNet101 is used as the backbone, while data-preprocessing steps, including data augmentation, are identical. We compare both networks and experiment with different fusion techniques in U-Net and with hyper-parameters for weighting the input channels for fusion in DeepLab V3+. We also evaluate the effect of pre-training on RGB and non-RGB data. The results show a minimally better performance of the DeepLab V3+ model compared to U-Net, while for the certain classes, such as vehicles, U-Net yields a slightly superior accuracy.

show abstract

Diagnosis of interproximal caries lesions with deep convolutional neural network in digital bitewing radiographs

Bayraktar

Ayan

2021

Clin Oral Invest

View full text Add to dashboard Cite

Objectives This study aimed to investigate the effectiveness of deep convolutional neural network (CNN) in the diagnosis of interproximal caries lesions in digital bitewing radiographs. Methods and materials A total of 1,000 digital bitewing radiographs were randomly selected from the database. Of these, 800 were augmented and annotated as “decay” by two experienced dentists using a labeling tool developed in Python programming language. The 800 radiographs were consisted of 11,521 approximal surfaces of which 1,847 were decayed (lesion prevalence for train data was 16.03%). A CNN model known as you only look once (YOLO) was modified and trained to detect caries lesions in bitewing radiographs. After using the other 200 radiographs to test the effectiveness of the proposed CNN model, the accuracy, sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), and area under the receiver operating characteristic curve (AUC) were calculated. Results The lesion prevalence for test data was 13.89%. The overall accuracy of the CNN model was 94.59% (94.19% for premolars, 94.97% for molars), sensitivity was 72.26% (75.51% for premolars, 68.71% for molars), specificity was 98.19% (97.43% for premolars, 98.91% for molars), PPV was 86.58% (83.61% for premolars, 90.44% for molars), and NPV was 95.64% (95.82% for premolars, 95.47% for molars). The overall AUC was measured as 87.19%. Conclusions The proposed CNN model showed good performance with high accuracy scores demonstrating that it could be used in the diagnosis of caries lesions in bitewing radiographs. Clinical significance Correct diagnosis of dental caries is essential for a correct treatment procedure. CNNs can assist dentists in diagnosing approximal caries lesions in bitewing radiographs.

show abstract

Multimodal Semantic Segmentation: Fusion of RGB and Depth Data in Convolutional Neural Networks

Cited by 7 publications

References 12 publications

Dental student application of artificial intelligence technology in detecting proximal caries lesions

Dental student application of artificial intelligence technology in detecting proximal caries lesions

Exploring fusion techniques in U-Net and DeepLab V3 architectures for multi-modal land cover classification

Diagnosis of interproximal caries lesions with deep convolutional neural network in digital bitewing radiographs

Contact Info

Product

Resources

About