In recent years, the lack of thermal images and the difficulty of thermal feature extraction have led to low accuracy and efficiency in the fault diagnosis of circuit boards using thermal images. To address the problem, this paper presents a simple and efficient intelligent fault diagnosis method combined with computer vision, namely the bag-of-SURF-features support vector machine (BOSF-SVM). Firstly, an improved BOF feature extraction based on SURF is proposed. The preliminary fault features of the abnormally hot components are extracted by the speeded-up robust features algorithm (SURF). In order to extract the ultimate fault features, the preliminary fault features are clustered into K clusters by K-means and substituted into the bag-of-features model (BOF) to generate a bag-of-SURF-feature vector (BOSF) for each image. Then, all of the BOSF vectors are fed into SVM to train the fault classification model. Finally, extensive experiments are conducted on two homemade thermal image datasets of circuit board faults. Experimental results show that the proposed method is effective in extracting the thermal fault features of components and reducing misdiagnosis and underdiagnosis. Also, it is economical and fast, facilitating savings in labour costs and computing resources in industrial production.