This study aims to build a system for detecting a driver’s internal state using body-worn sensors. Our system is intended to detect inattentive driving that occurs during long-term driving on a monotonous road, such as a high-way road. The inattentive state of a driver in this study is an absent-minded state caused by a decrease in driver vigilance levels due to fatigue or drowsiness. However, it is difficult to clearly define these inattentive states because it is difficult for the driver to recognize when they fall into an absent-minded state. To address this problem and achieve our goal, we have proposed a detection algorithm for inattentive driving that not only uses a heart rate sensor, but also uses body-worn inertial sensors, which have the potential to detect driver behavior more accurately and at a much lower cost. The proposed method combines three detection models: body movement, drowsiness, and inattention detection, based on an anomaly detection algorithm. Furthermore, we have verified the accuracy of the algorithm with the experimental data for five participants that were measured in long-term and monotonous driving scenarios by using a driving simulator. The results indicate that our approach can detect both the inattentive and drowsiness states of drivers using signals from both the heart rate sensor and accelerometers placed on wrists.