This paper is devoted to the development of a deep learning- (DL-) based model to detect crack fractures on concrete surfaces. The developed model for the classification of images was based on a DL Convolutional Neural Network (CNN). To train and validate the CNN model, a database containing 40,000 images of concrete surfaces (with and without cracks) was collected from the available literature. Several conditions on the concrete surfaces were taken into account such as illumination and surface finish (i.e., exposed, plastering, and paint). Various error measurement criteria such as accuracy, precision, recall, specificity, and F1-score were employed for accessing the quality of the developed model. Results showed that for the training dataset (50% of the database), the precision, recall, specificity, F1-score, and accuracy were 99.5%, 99.8%, 99.5%, 99.7%, and 99.7%, respectively. On the other hand, for the validating dataset, the precision, recall, specificity, F1-score, and accuracy are 96.5%, 98.8%, 96.6%, 97.7%, and 97.7%, respectively. Thus, the developed CNN model may be considered valid because it performs the classification of cracks well using the testing data. It is also confirmed that the developed DL-based model was robust and efficient, as it can take into account different conditions on the concrete surfaces. The CNN model developed in this study was compared with other works in the literature, showing that the CNN model could improve the accuracy of image classification, in comparison with previously published results. Finally, in further work, such model could be combined with Unmanned Aerial Vehicles (UAVs) to increase the productivity of concrete infrastructure inspection.