With development of the heavy-haul railway, the increased axle load and traction weight bring a significant challenge for the service performance and safety maintenance of the railway track. Conducting defect recognition on concrete sleepers and ballast using big data is vital. This paper focused on the detection of absent sleeper support in a ballasted track with an emphasis on the integration of model-based and data-driven methods. To this end, a mathematical model consisting of the wagon, track and wheel–rail contact subsystems was first established to acquire the necessary raw data for the data-driven method, in which the wagon was regarded as a 47-degree-of-freedom multi-body subsystem, and the track was treated as a multi-layer discrete-elastic support beam subsystem with absent sleeper support. Then, an architectural hierarchy of a three-layer convolutional neural network (TLCNN) was developed, which includes three convolutional layers and two pooling layers, and a method for reconstructing one-dimensional sleeper vertical displacement to a two-dimensional time–space matrix was also proposed. Thirdly, verification was carried out by comparing the simulation and experimental results to illustrate the accuracy and reliability of the mathematical model, and the dynamic behaviour of the track with absent sleeper support was investigated. Lastly, the established TLCNN was used to train the raw data of the sleeper vertical displacement and detect the existence of absent sleeper support. Results show that the integration of model-based and data-driven methods was a reliable and effective approach for the detection of absent sleeper support. The proposed TLCNN can acquire and extract robust characteristics in a noisy environment. To handle more complex recognition tasks and further improve performance, deeper CNN models and larger sample sizes should be preferentially considered in practical applications.