For autonomous vehicles and intelligent connected vehicles, the real-time recognition of risky drivers can play an important role in traffic accident prevention. However, the external environment substantially impacts driving behavior and driving risk and is usually costly to acquire. Existing risky driver recognition models often ignore external environment information or assume this information is given. We propose two hierarchical two-layer context-aware machine learning structures. The first layer can speculate external context, for example, traffic states. The second layer recognizes risky drivers based on the contextual information speculated from the first layer. The German Highway Drone Dataset is used to establish risky driver recognition and traffic state recognition models. Rear-end collision risk and side collision risk are evaluated for each vehicle. Drivers with high collision risk are labeled as risky drivers. By analyzing vehicle trajectory data from three traffic states: free-flow, saturated, and congested, we find that traffic states have a significant influence on vehicle's longitudinal speed, lateral speed, longitudinal acceleration/deceleration, and collision risk. Six classifiers, including SVM, KNN, RF, Adaboost, Extra trees, and XGBoost, are applied to train recognition models. Results show that the proposed structures can significantly improve model's ability to recognize risky drivers.