Concrete pavement defects are an important indicator reflecting the safety status of pavement. However, it is difficult to accurately detect the concrete pavement cracks due to the complex concrete pavement environment, such as uneven illumination, deformation and potential shadows, etc. In order to solve these problems, we propose the crack detection algorithm of concrete pavement with convolutional neural network. Firstly, our method is used to classify cracks first and detect the classified crack images, different deep learning models are used in these two parts to achieve different functions. Secondly, in the crack classification section, in view of the low proportion of effective concrete pavement crack images in the mass images collected by crack detection vehicle, the output dimension of FC2 layer of LeNet-5 model is modified before crack detection. It can accurately identify the concrete pavement cracks from several types of disturbance characteristics by training the classification model. Finally, in order to improve the efficiency of crack detection, the algorithm scales the network model horizontally and accesses the convolution layer with the kernel size of 1 × 1, 3 × 3. Experiments show that the F 1 of our algorithm reaches to 0.896 in CFD dataset. Compared with VGG16, U-Net and Percolation, it is 25.2%, 2.8%, 39.1% improvement of F 1 respectively. For Cracktree200 dataset, the F 1 is 0.892. Compared with VGG16, U-Net and Percolation, it is 50.3%, 16.6%, 68.9% improvement of F 1 respectively. For DeepCrack dataset, the F 1 is 0.901. Compared with VGG16, U-Net and Percolation, it is 53%, 5.2%, 52.2% improvement of F 1 respectively.