In this paper, a vision-based physiological signal measurement system is proposed to instantly measure driver fatigue. A remote photoplethysmography (rPPG) signal is a type of physiological signal measured by a camera without any contact device, and it also retains the characteristics of the PPG, which is useful to evaluate fatigue. To solve the inconvenience caused by the traditional contact-based physiological fatigue detection system and to improve the accuracy, the system measures both the motional and physiological information by using one image sensor. In a practical application, the environmental noise would affect the measured signal, and therefore, we performed a preprocessing step on the signal to extract a clear signal. The experiment was designed in collaboration with Taipei Medical University, and a questionnaire-based method was used to define fatigue. The questionnaire that could directly react to the feeling of the subject was treated as our ground truth. The evaluated correlation was 0.89 and the root mean square error was 0.65 for ten-fold cross-validation on the dataset. The trend of driver fatigue could be evaluated without a contact device by the proposed system. This advantage ensures the safety of the driver and reliability of the system.INDEX TERMS Fatigue monitoring, remote photoplethysmography, biomedical monitoring, image sequence analysis.