By blending computer-aided medical systems with cutting-edge privacy technologies, healthcare providers can deliver more personalized, effective care while maintaining the highest data security standards and patient trust. The challenge of dental segmentation in computer vision, a task focused on accurately outlining dental structures in images, traditional methods, particularly convolution neural networks (CNNs), didn't reach high accuracy in this area due to suboptimal performance and computational inefficiency. The goal of image segmentation is to group pixels on the basis of their visual properties, such as color, texture, intensity, or spatial proximity, to identify and delineate the boundaries of distinct objects or regions within the image. In this paper, You Only Look Once (YOLOv8) algorithm is improved to segment teeth with high accuracy and high execution speed. The increase in the number of layers of YOLOv8 relied upon, as the accuracy of the algorithm segmentation depends on the number of layers used to extract features from the image (backbone) and the number of layers of the head (prediction). In addition, the size of the layers is decreased to increase the execution speed. The novelty of this paper is the proposed YOLOv8 model in addition to the Proposed Activation Function (PAF). The dataset (top view) used was taken from a dental clinic where 526 images were taken of dental and different patients. The best accuracy reached 99.561% when the enhanced YOLOv8 segmentation model was applied to the dental dataset. It can be concluded that the improved model of the YOLOv8 algorithm has increased the accuracy of dental segmentation compared to previous research because it relies on a proposed PAF that increases the difference between the features extracted from the layers of the proposed model which makes it able to distinguish between teeth and surrounding parts significantly.
