The defects in the welds of energy pipelines have significantly influenced their safe operation. The inefficient and inaccurate detection of the defects may give rise to catastrophic accidents. Ultrasonic phased array inspection is an important means of ensuring pipeline safety. The total focusing method (TFM), using ultrasonic phased arrays, has become widely used in recent years in non-destructive evaluation (NDE). However, manual defect recognition of TFM images is seen to lack accuracy and robustness, arising from deficiency of practical experience. In this paper, the automated classification of different defects from TFM images is studied with a view to facilitate inspection efficacy. By experimentally implementing the TFM approach on a bespoke specimen, the images corresponding to crack-like defects and pore-like defects were employed to investigate the effectiveness of four different machine learning models (known as Support Vector Machine, CART Decision tree, K Nearest Neighbors, Naive Bayes) containing data augmentation, feature extraction and defect classification. The results suggested that the accuracy of defect classification using the HOG-Poly-SVM algorithm was 93%, which outperformed the results from other algorithms. The advantage of the HOG-Poly-SVM algorithm used in defect classification of ultrasonic phased array TFM data is discussed by conducting ten-fold cross validation and other evaluation metrics. In this paper, in order to improve the efficiency of detecting pipeline defects in the future, and for testing test blocks simulating buried pipelines containing defects, we proposed, for the first time, that ultrasonic phased-array TFM imaging results in small object detection images, and found that the SVM algorithm was applicable to ultrasonic phased array TFM imaging, providing a research method and ideas for the use of artificial intelligence in industrial non-destructive testing.