The crack ratio is one of the indices used to quantitatively evaluate the soundness of asphalt pavement. However, since the inspection of pavement requires much labor and cost, automatic inspection of pavement damage by image analysis is required in order to reduce the burden of such work. In this study, a system was constructed that automatically detects and evaluates cracks from images of pavement using a convolutional neural network, a kind of deep learning. The most novel aspect of this study is that the accuracy was recursively improved through retraining the convolutional neural network (CNN) by collecting images which had previously been incorrectly analyzed. Then, study and implementation were conducted of a system for plotting the results in a GIS. In addition, an experiment was carried out applying this system to images actually taken from an MMS (mobile mapping system), and this confirmed that the system had high crack evaluation performance.