Metal additive manufacturing (AM) is a disruptive production technology, widely adopted in innovative industries that revolutionizes design and manufacturing. The interest in quality control of AM systems has grown substantially over the last decade, driven by AM’s appeal for intricate, high-value, and low-volume production components. Geometry-dependent process conditions in AM yield unique challenges, especially regarding quality assurance. This study contributes to the development of machine learning models to enhance in-process monitoring and control technology, which is a critical step in cost reduction in metal AM. As the part is built layer upon layer, the features of each layer have an influence on the quality of the final part. Layer-wise in-process sensing can be used to retrieve condition-related features and help detect defects caused by improper process conditions. In this work, layer-wise monitoring using optical tomography (OT) imaging was employed as a data source, and a machine-learning (ML) technique was utilized to detect anomalies that can lead to defects. The major defects analyzed in this experiment were gas pores and lack of fusion defects. The Random Forest Classifier ML algorithm is employed to segment anomalies from optical images, which are then validated by correlating them with defects from computerized tomography (CT) data. Further, 3D mapping of defects from CT data onto the OT dataset is carried out using the affine transformation technique. The developed anomaly detection model’s performance is evaluated using several metrics such as confusion matrix, dice coefficient, accuracy, precision, recall, and intersection-over-union (IOU). The k-fold cross-validation technique was utilized to ensure robustness and generalization of the model’s performance. The best detection accuracy of the developed anomaly detection model is 99.98%. Around 79.40% of defects from CT data correlated with the anomalies detected from the OT data.