High-speed railway administrations are particularly concerned about safety and comfort issues, which are sometimes threatened by the differential deformation of substructures. Existing deformation-monitoring techniques are impractical for covering the whole range of a railway line at acceptable costs. Fortunately, the information about differential substructure deformation is contained in the dynamic inspection data of longitudinal level from comprehensive inspections trains. In order to detect potential differential deformations, an identification method, combining digital filtering, a convolutional neural network and infrastructure base information, is proposed. In this method, a low-pass filter is designed to remove short-waveband components of the longitudinal level. Then, a one-dimensional convolutional neural network is constructed to serve as a feature extractor from local longitudinal-level waveforms, and a binary classifier of potential differential deformations in place of the visual judgement of humans with profound expertise. Finally, the infrastructure base information is utilized to further classify the differential deformations into several types, according to the positional distribution of the substructures. The inspection data of four typical high-speed railways are selected to train and test the method. The results show that the convolutional neural network can identify differential substructure-deformations, with the precision, recall, accuracy and F1 score all exceeding 98% on the test data. In addition, four types of deformation can be further classified with the support of infrastructure base information. The proposed method can be used for directly locating adverse substructure deformations, and is also becoming a promising addition to existing deformation monitoring methods.