Railway wheelsets are the key to ensuring the safe operation of trains. To achieve zero-defect production, railway equipment manufacturers must strictly control every link in the wheelset production process. The press-fit curve output by the wheelset assembly machine is an essential indicator of the wheelset’s assembly quality. The operators will still need to manually and individually recheck press-fit curves in our practical case. However, there are many uncertainties in the manual inspection. For example, subjective judgment can easily cause inconsistent judgment results between different inspectors, or the probability of human misinterpretation can increase as the working hours increase. Therefore, this study proposes an intelligent railway wheelset inspection system based on deep learning, which improves the reliability and efficiency of manual inspection of wheelset assembly quality. To solve the severe imbalance in the number of collected images, this study establishes a predicted model of press-fit quality based on a deep Siamese network. Our experimental results show that the precision measurement is outstanding for the testing dataset contained 3863 qualified images and 28 unqualified images of press-fit curves. The proposed system will serve as a successful case of a paradigm shift from traditional manufacturing to digital manufacturing.