Brain fatigue is often associated with inattention, mental retardation, prolonged reaction time, decreased work efficiency, increased error rate, and other problems. In addition to the accumulation of fatigue, brain fatigue has become one of the important factors that harm our mental health. Therefore, it is of great significance to explore the practical and accurate brain fatigue detection method, especially for quantitative brain fatigue evaluation. In this study, a biomedical signal of ballistocardiogram (BCG), which does not require direct contact with human body, was collected by optical fiber sensor cushion during the whole process of cognitive tasks for 20 subjects. The heart rate variability (HRV) was calculated based on BCG signal. Machine learning classification model was built based on random forest to quantify and recognize brain fatigue. The results showed that: Firstly, the heart rate obtained from BCG signal was consistent with the result displayed by the medical equipment, and the absolute difference was less than 3 beats/min, and the mean error is 1.30 ± 0.81 beats/min; secondly, the random forest classifier for brain fatigue evaluation based on HRV can effectively identify the state of brain fatigue, with an accuracy rate of 96.54%; finally, the correlation between HRV and the accuracy was analyzed, and the correlation coefficient was as high as 0.98, which indicates that the accuracy can be used as an indicator for quantitative brain fatigue evaluation during the whole task. The results suggested that the brain fatigue quantification evaluation method based on the optical fiber sensor cushion and machine learning can carry out real-time brain fatigue detection on the human brain without disturbance, reduce the risk of human accidents in human–machine interaction systems, and improve mental health among the office and driving personnel.