Background and purpose: Accurate instance segmentation of teeth in panoramic dental X-rays is a challenging task due to variations in tooth morphology and overlapping regions. In this study, we propose a new algorithm, for instance, segmentation of the different teeth in panoramic dental X-rays. Methods: An instance segmentation model was trained using the architecture of a Mask Region-based Convolutional Neural Network (Mask-RCNN). The data for the training, validation, and testing were taken from the Tuft dental database (1000 panoramic dental radiographs). The number of the predicted label was 52 (20 deciduous and 32 permanent). The size of the training, validation, and test sets were 760, 190, and 70 images, respectively, and the split was performed randomly. The model was trained for 300 epochs, using a batch size of 10, a base learning rate of 0.001, and a warm-up multistep learning rate scheduler (gamma = 0.1). Data augmentation was performed by changing the brightness, contrast, crop, and image size. The percentage of correctly detected teeth and Dice in the test set were used as the quality metrics for the model. Results: In the test set, the percentage of correctly classified teeth was 98.4%, while the Dice score was 0.87. For both the left mandibular central and lateral incisor permanent teeth, the Dice index result was 0.91 and the accuracy was 100%. For the permanent teeth right mandibular first molar, mandibular second molar, and third molar, the Dice indexes were 0.92, 0.93, and 0.78, respectively, with an accuracy of 100% for all three different teeth. For deciduous teeth, the Dice indexes for the right mandibular lateral incisor, right mandibular canine, and right mandibular first molar were 0.89, 0.91, and 0.85, respectively, with an accuracy of 100%. Conclusions: A successful instance segmentation model for teeth identification in panoramic dental X-ray was developed and validated. This model may help speed up and automate tasks like teeth counting and identifying specific missing teeth, improving the current clinical practice.