In recent years, with the rapid increase in coverage and lines, security maintenance has become one of the top concerns with regard to railway transportation in China. As the key transportation infrastructure, the railway turnout system (RTS) plays a vital role in transportation, which will cause incalculable losses when accidents occur. The traditional fault-diagnosis and maintenance methods of the RTS are no longer applicable to the growing amount of data, so intelligent fault diagnosis has become a research hotspot. However, the key challenge of RTS intelligent fault diagnosis is to effectively extract the deep features in the signal and accurately identify failure modes in the face of unbalanced datasets. To solve the above two problems, this paper focuses on unbalanced data and proposes a fault-diagnosis method based on an improved autoencoder and data augmentation, which realizes deep feature extraction and fault identification of unbalanced data. An improved autoencoder is proposed to smooth the noise and extract the deep features to overcome the noise fluctuation caused by the physical characteristics of the data. Then, synthetic minority oversampling technology (SMOTE) is utilized to effectively expand the fault types and solve the problem of unbalanced datasets. Furthermore, the health state is identified by the Softmax regression model that is trained with the balanced characteristics data, which improves the diagnosis precision and generalization ability. Finally, different experiments are conducted on a real dataset based on a railway station in China, and the average diagnostic accuracy reaches 99.13% superior to other methods, which indicates the effectiveness and feasibility of the proposed method.