As an important part of traditional Chinese architecture, Fuzhou’s ancient houses have unique cultural and historical value. However, over time, environmental factors such as efflorescence and plant growth have caused surface damage to their gray brick walls, leading to a decline in the quality of the buildings’ structure and even posing a threat to the buildings’ safety. Traditional damage detection methods mainly rely on manual labor, which is inefficient and consumes a lot of human resources. In addition, traditional non-destructive detection methods, such as infrared imaging and laser scanning, often face difficulty in accurately identifying specific types of damage, such as efflorescence and plant growth, on the surface of gray bricks and are easily hampered by diverse surface features. This study uses the YOLOv8 machine learning model for the automated detection of two common types of damage to the gray brick walls of Fuzhou’s ancient houses: efflorescence and plant growth. We establish an efficient gray brick surface damage detection model through dataset collection and annotation, experimental parameter optimization, model evaluation, and analysis. The research results reveal the following. (1) Reasonable hyperparameter settings and model-assisted annotation significantly improve the detection accuracy and stability. (2) The model’s average precision (AP) is improved from 0.30 to 0.90, demonstrating good robustness in detecting complex backgrounds and high-resolution real-life images. The F1 value of the model’s gray brick detection efficiency is improved (classification model performance index) from 0.22 to 0.77. (3) The model’s ability to recognize the damage details of gray bricks under high-resolution conditions is significantly enhanced, demonstrating its ability to cope with complex environments. (4) The simplified data enhancement strategy effectively reduces the feature extraction interference and enhances the model’s adaptability in different environments.