Understanding and mitigating drivers' negative emotions, stress levels, and anxiety is of high importance for decreasing accident rates, enhancing road safety, and providing a healthy lifestyle to the community of drivers. While detecting drivers' stress and negative emotions can significantly help with this goal, understanding what might be associated with increases in drivers' negative emotions and high stress level, might better help with planning interventions. While studies have provided significant insight into detecting drivers' emotions and stress levels; not many studies focused on the reasons behind changes in stress levels and negative emotions. In this study, by using a naturalistic driving study database, we analyze the changes in the driving scene, including road objects and the dynamical relationship between the ego vehicle and the lead vehicle with respect to changes in drivers' psychophysiological metrics (i.e., heart rate (HR) and facial expressions). We find that different road objects might be associated with varying levels of increase in drivers' HR as well as different proportions of negative facial emotions detected through computer vision. Our results indicate that larger vehicles on the road, such as trucks and buses, are associated with the highest amount of increase in drivers' HR as well as negative emotions. Additionally, we provide evidence that shorter distances to the lead vehicle in naturalistic driving, as well as the higher standard deviation in the distance, might be associated with a higher number of abrupt increases in drivers' HR, depicting a possible increase in stress level. Lastly, our results indicate more positive emotions, lower facial engagement, and a lower abrupt increase in HR at a higher speed of driving, which often happens in highway driving. Our results collectively show that driving at higher speeds happening in highways by avoiding certain road objects might be a better fit for keeping drivers in a calmer, more positive state. This research lays the ground for designing human-centered vehicles, urban environments, and services that can understand the level to which each road object and environment might affect drivers' and passengers' wellbeing.