Ensuring fire safety is essential to protect life and property, but modern infrastructure and complex settings require advanced fire detection methods. Traditional object detection systems, often reliant on manual feature extraction, may fall short, and while deep learning approaches are powerful, they can be computationally intensive, especially for real-time applications. This paper proposes a novel smoke and fire detection method based on the YOLOv8n model with several key architectural modifications. The standard Complete-IoU (CIoU) box loss function is replaced with the more robust Wise-IoU version 3 (WIoUv3), enhancing predictions through its attention mechanism and dynamic focusing. The model is streamlined by replacing the C2f module with a residual block, enabling targeted feature extraction, accelerating training and inference, and reducing overfitting. Integrating generalized efficient layer aggregation network (GELAN) blocks with C2f modules in the neck of the YOLOv8n model further enhances smoke and fire detection, optimizing gradient paths for efficient learning and high performance. Transfer learning is also applied to enhance robustness. Experiments confirmed the excellent performance of ESFD-YOLOv8n, outperforming the original YOLOv8n by 2%, 2.3%, and 2.7%, with a mean average precision (mAP@0.5) of 79.4%, precision of 80.1%, and recall of 72.7%. Despite its increased complexity, the model outperforms several state-of-the-art algorithms and meets the requirements for real-time fire and smoke detection.