PurposeFire smoke detection in petrochemical plant can prevent fire and ensure production safety and life safety. The purpose of this paper is to solve the problem of missed detection and false detection in flame smoke detection under complex factory background.Design/methodology/approachThis paper presents a flame smoke detection algorithm based on YOLOv5. The target regression loss function (CIoU) is used to improve the missed detection and false detection in target detection and improve the model detection performance. The improved activation function avoids gradient disappearance to maintain high real-time performance of the algorithm. Data enhancement technology is used to enhance the ability of the network to extract features and improve the accuracy of the model for small target detection.FindingsBased on the actual situation of flame smoke, the loss function and activation function of YOLOv5 model are improved. Based on the improved YOLOv5 model, a flame smoke detection algorithm with generalization performance is established. The improved model is compared with SSD and YOLOv4-tiny. The accuracy of the improved YOLOv5 model can reach 99.5%, which achieves a more accurate detection effect on flame smoke. The improved network model is superior to the existing methods in running time and accuracy.Originality/valueAiming at the actual particularity of flame smoke detection, an improved flame smoke detection network model based on YOLOv5 is established. The purpose of optimizing the model is achieved by improving the loss function, and the activation function with stronger nonlinear ability is combined to avoid over-fitting of the network. This method is helpful to improve the problems of missed detection and false detection in flame smoke detection and can be further extended to pedestrian target detection and vehicle running recognition.