Fatigue driving (FD) is one of the main causes of traffic accidents. Traditionally, machine learning technologies such as back propagation neural network (BPNN) and support vector machine (SVM) are popularly used for fatigue driving detection. However, the BPNN exhibits slow convergence speed and many adjustable parameters, while it is difficult to train large-scale samples in the SVM. In this paper, we develop extreme learning machine (ELM)-based FD detection method to avoid the above disadvantages. Further, since the randomness of the weight and biases between the input layer and the hidden layer of the ELM will influence its generalization performance, we further apply a differential evolution ELM (DE-ELM) method to the analysis of the driver’s respiration and heartbeat signals, which can effectively judge the driver fatigue state. Moreover, not only will the Doppler radar and smart bracelet be used to obtain the driver respiration and heartbeat signals, but also the sample database required for the experiment will be established through extensive signal collections. Experimental results show that the DE-ELM has a better performance on driver’s fatigue level detection than the traditional ELM and SVM.