Due to its wide monitoring range and low cost, visual-based fire detection technology is commonly used for fire detection in open spaces. However, traditional fire detection algorithms have limitations in terms of accuracy and speed, making it challenging to detect fires in real time. These algorithms have poor anti-interference ability against fire-like objects, such as emissions from factory chimneys, clouds, etc. In this study, we developed a fire detection approach based on an improved YOLOv5 algorithm and a fire detection dataset with fire-like objects. We added three Convolutional Block Attention Modules (CBAMs) to the head network of YOLOv5 to improve its feature extraction ability. Meanwhile, we used the C2f module to replace the original C2 module to capture rich gradient flow information. Our experimental results show that the proposed algorithm achieved a mAP@50 of 82.36% for fire detection. In addition, we also conducted a comparison test between datasets with and without labeling information for fire-like objects. Our results show that labeling information significantly reduced the false-positive detection proportion of fire-like objects incorrectly detected as fire objects. Our experimental results show that the CBAM and C2f modules enhanced the network’s feature extraction ability to differentiate fire objects from fire-like objects. Hence, our approach has the potential to improve fire detection accuracy, reduce false alarms, and be more cost-effective than traditional fire detection methods. This method can be applied to camera monitoring systems for automatic fire detection with resistance to fire-like objects.