In this paper, the candidate region network in the Fast R-CNN algorithm is used to obtain the feature values of surface damage of ancient buildings, and the regression algorithm is applied to the target bounding box after the features are successfully obtained, which can more accurately complete the task of detecting and locating the damage on the surface of buildings. The three-dimensional laser scanner and total station are used to explore the surrounding environment of the buildings on the site, and the sketch is drawn according to the research and analysis results combined with the exploration to design a digital archive of the cultural heritage of the ancient buildings. Aiming at the ambiguity of the assessment system of the integrity of ancient buildings, we constructed the assessment system of the cultural heritage of ancient buildings with the fuzzy affiliation of the data of the integrity of ancient buildings and multilevel fusion assessment and carried out an example analysis of the research on the protection of the cultural heritage of ancient buildings in the era of smart media. The results show that when the number of iterations is 15000, the mAP value is 0.9411 (where 1 represents the maximum AP value), which is higher than the training results with the number of iterations of 20000 and 25000, i.e., the Faster R-CNN training model has good applicability for the autonomous visual detection of the damage on the surface of the ancient buildings. “Q9 I will not damage the ancient buildings in Zhuge Village” has the highest mean value of 4.71 and the highest approval rate of 99.1, and the standard deviation of each question item is small, i.e., it indicates that there is a high degree of consistency in the positive attitudes of residents towards the protection of the cultural heritage of ancient buildings. This study contributes new ideas to the protection of ancient architectural, cultural heritage and tourism development, which is crucial for advancing the research field.