The study of the robust fatigue feature learning method for the driver’s operational behavior is of great significance for improving the performance of the real-time detection system for driver’s fatigue state. Aiming at how to extract more abstract and deep features in the driver’s direction operation data in the robust feature learning, this article constructs a fuzzy recurrent neural network model, which includes input layer, fuzzy layer, hidden layer, and output layer. The steering-wheel direction sensing time series sends the time series to the input layer through a fixed time window. After the fuzzification process, it is sent to the hidden layer to share the weight of the hidden layer, realize the memorization of the fatigue feature, and improve the feature depth capability of the steering wheel angle time sequence. The experimental results show that the proposed model achieves an average recognition rate of 87.30% in the fatigue sample database of real vehicle conditions, which indicates that the model has strong robustness to different subjects under real driving conditions. The model proposed in this article has important theoretical and engineering significance for studying the prediction of fatigue driving under real driving conditions.