The magnetic tile image has the characteristics of uneven illumination, complex surface texture, and low contrast. Aiming at the problem that the traditional defect detection algorithm is difficult to accurately identify the defects, and the deep learning algorithm is difficult to balance the classification accuracy and the size of the speed model, a defect classification algorithm based on attention-based EfficientNet is proposed. The algorithm first enhances the network’s spatial and location information for image features by integrating the Convolutional Block Attention Module, and improves the network’s ability to identify defects. Then, on this basis, Criss-Cross Attention is added to the network, so that the network can better the context information of the horizontal and vertical cross of image features, so that each pixel can finally capture the full image dependency of all pixels. Experimental results show that the algorithm has higher classification accuracy than EfficientNet-B0, reached 99.11%, and has a better balance between accuracy, speed and model size than other classification models.