This work provides an innovative endodontic instrument fault detection methodology during root canal treatment (RCT). Sometimes, an endodontic instrument is prone to fracture from the tip, for causes uncertain the dentist’s control. A comprehensive assessment and decision support system for an endodontist may avoid several breakages. This research proposes a machine learning and artificial intelligence-based approach that can help to diagnose instrument health. During the RCT, force signals are recorded using a dynamometer. From the acquired signals, statistical features are extracted. Because there are fewer instances of the minority class (i.e. faulty/moderate class), oversampling of datasets is required to avoid bias and overfitting. Therefore, the synthetic minority oversampling technique (SMOTE) is employed to increase the minority class. Further, evaluating the performance using the machine learning techniques, namely Gaussian Naïve Bayes (GNB), quadratic support vector machine (QSVM), fine k-nearest neighbor (FKNN), and ensemble bagged tree (EBT). The EBT model provides excellent performance relative to the GNB, QSVM, and FKNN. Machine learning (ML) algorithms can accurately detect endodontic instruments’ faults by monitoring the force signals. The EBT and FKNN classifier is trained exceptionally well with an area under curve values of 1.0 and 0.99 and prediction accuracy of 98.95 and 97.56%, respectively. ML can potentially enhance clinical outcomes, boost learning, decrease process malfunctions, increase treatment efficacy, and enhance instrument performance, contributing to superior RCT processes. This work uses ML methodologies for fault detection of endodontic instruments, providing practitioners with an adequate decision support system.