This paper introduces a novel approach to pavement material crack detection, classification, and segmentation using advanced deep learning techniques, including multi-scale feature aggregation and transformer-based attention mechanisms. The proposed methodology significantly enhances the model’s ability to handle varying crack sizes, shapes, and complex pavement textures. Trained on a dataset of 10,000 images, the model achieved substantial performance improvements across all tasks after integrating transformer-based attention. Detection precision increased from 88.7% to 94.3%, and IoU improved from 78.8% to 93.2%. In classification, precision rose from 88.3% to 94.8%, and recall improved from 86.8% to 94.2%. For segmentation, the Dice Coefficient increased from 80.3% to 94.7%, and IoU for segmentation advanced from 74.2% to 92.3%. These results underscore the model’s robustness and accuracy in identifying pavement cracks in challenging real-world scenarios. This framework not only advances automated pavement maintenance but also provides a foundation for future research focused on optimizing real-time processing and extending the model’s applicability to more diverse pavement conditions.