Mode recognition is a basic task to interpret the behavior of multi-functional radar. The existing methods need to train complex and huge neural networks to improve the recognition ability, and it is difficult to deal with the mismatch between the training set and the test set. In this paper, a learning framework based on residual neural network (ResNet) and support vector machine (SVM) is designed, to solve the problem of mode recognition for non-specific radar, called multi-source joint recognition framework (MSJR). The key idea of the framework is to embed the prior knowledge of radar mode into the machine learning model, and combine the manual intervention and automatic extraction of features. The model can purposefully learn the feature representation of the signal on the working mode, which weakens the impact brought by the mismatch between training and test data. In order to solve the problem of difficult recognition under signal defect conditions, a two-stage cascade training method is designed, to give full play to the data representation ability of ResNet and the high-dimensional feature classification ability of SVM. Experiments show that the average recognition rate of the proposed model, with embedded radar knowledge, is improved by 33.7% compared with the purely data-driven model. Compared with other similar state-of-the-art reported models, such as AlexNet, VGGNet, LeNet, ResNet, and ConvNet, the recognition rate is increased by 12%. Under the condition of 0–35% leaky pulses in the independent test set, MSJR still has a recognition rate of more than 90%, which also proves its effectiveness and robustness in the recognition of unknown signals with similar semantic characteristics.