Driver distraction behavior causes a large number of traffic accidents every year, resulting in economic losses and injuries. Currently, the driver still plays an important role in the driving and control of the vehicle due to the low level of vehicle automation and the immature development of autonomous driving. Therefore, it is vital to research distraction detection for drivers. However, in realistic driving scenarios with uncertain information, they are still some challenges in efficient and accurate driver distraction detection. In this paper, an improved deep learning model based on attention mechanisms and bi-directional feature pyramid networks (BiFPN) is proposed to identify driver distractions. Firstly, an improved data augmentation strategy is introduced to increase the data diversity to enhance the generalization capability of the model. Secondly, the squeeze-and-excitation (SE) attention mechanism layer is used after the C3 module of the original backbone network to enhance the important feature information and suppress the minor feature information. Finally, the BiFPN module is introduced into the neck network to better achieve multi-scale feature fusion without increasing the calculation amount too much. The experimental results show that the method proposed in this paper has an average mean accuracy rate (mAP) of 0.956 on the test set. Compared to the original model the mAP has improved by 13.2%. The detection speed of the model is 71 frames per second, and the memory occupation is 15.9 MB. This method has the advantages of high recognition accuracy, fast detection speed, and small memory occupation of the model, which are important for achieving engineering deployment.