Multi-feature fusion has been widely used to enhance recognition accuracy for different health stages of rail, which may lead to high-dimensionality and information redundancy of signal. In addition, conventional supervised methods require plenty of labeled samples with class information, which can lead to significant time and economic costs. In order to improve the effectiveness of the electromagnetic acoustic emission (EMAE) technique in rail crack defect recognition, a novel method including multi-feature fusion based on weakly supervised learning and recognition threshold construction, is proposed in this paper. First, a mechanism contains of multi-feature extraction and feature selection, is developed to fully reflect the information of different health stages of rail and avoid interference caused by the ineffective features. Then, the effective features and a novel weakly unsupervised label are input into the self-normalizing convolutional neural network and long short-term memory (SCNN-LSTM) model to construct the rail health indicator (RHI). Finally, the recognition threshold is calculated by the characteristics of RHI, to achieve crack recognition automatically. Furthermore, the experimental results under different working conditions demonstrate that the proposed method achieves higher recognition performance than other existing methods in rail crack defect recognition.