A large percentage of traffic accidents are due to human errors. Driving behavior and driving stress influence the probability of making these mistakes. Both are influenced by multiple factors, among which might be elements such as age, gender, sleeping hours, or working hours. The objective of this paper is to study, in a real scenario and without forcing the driver’s state, the relationship between driving behavior, driving stress, and these elements. Furthermore, we aim to provide guidelines to improve driving assistants. In this study, we used 1050 driving samples obtained from 35 volunteers. The driving samples correspond to regular commutes from home to the workplace. ANOVA and ANCOVA tests were carried out to check if there are significant differences in the four factors analyzed. Although the results show that driving behavior and driving stress are affected by gender, age, and sleeping hours, the most critical variable is working hours. Drivers with long working days suffer significantly more driving stress compared to other drivers, with the corresponding effect on their driving style. These drivers were the worst at maintaining the safety distance.