Traffic accidents occur frequently in Internet of Things (IoT) safety system. Traffic accidents are largely caused by drivers’ unsafe driving behaviors in the process of driving. Aiming at the problem of low safety of real-time warning in driving, this paper proposes a model to detect driver behavior. Firstly, according to the driver target detection for positioning, combined with the Pose Estimation to identify the driver in the process of driving a variety of driving behaviors, at the same time, a rating model is built to score drivers’ driving behaviors. Then, by integrating the driver behavior model and evaluation rules, the system can give timely and active warning when the driver makes unsafe behavior in the process of driving. Finally, in the V2X scenario, feedback and presentation are given to users in the form of points. The experimental results show that, in the scenario of Internet of vehicles, the driving behavior rating model can well analyze and evaluate drivers’ driving behaviors, so that drivers can more accurately understand their abnormal driving behaviors and driving scores, which plays a significant role in IoT safety management.