<p><strong>Abstract.</strong> Outdoor stone cultural properties are continuously affected by the external environment such as wind, rain, and earthquakes. These cause damage to the cultural properties by not only threatening structural stability but also damaging the aesthetic value. Quick detection of these damages is important to enable appropriate preservation treatment in terms of cultural property conservation management. Even though conventional manual damage detection methods are widely used, they are limited by manpower, cost, and other external conditions. In this paper, we propose a system that automatically detects and classifies damage occurring in cultural properties using deep-learning technique to settle these drawbacks. In detail, the damages are classified into four types (i.e., crack, loss, detachment, biological colonization) based on Faster region-based convolutional neural network (R-CNN) algorithm. In addition, we construct an image dataset of stone damage, which is collected by the regular report of the National Designated Cultural Property in 2017 conducted by the Cultural Heritage Administration of S. Korea, and augment its dataset to enhance damage detection performance. From the experiment conducted, we achieved an average confidence score of 94.6&thinsp;% or more on the 20 test images.</p>