Research on fire detection based on deep learning has been increasing lately, and current models differ in accuracy and computational complexity. To reduce the loss of semantic information and improve prediction accuracy, this paper combines dynamic threshold grayscale segmentation and residual network transfer learning. An effective training data extension strategy integrates the experimental data of a real fire platform and a forest-image-based fire database, where the experimental data of the real fire platform are shot based on different lighting and backgrounds. It has been proven that this is a good solution that can effectively solve the problem of training robust deep models relying on datasets with large diversity, which can improve the overall performance of the training model. Then, we introduce a network architecture based on dynamic threshold grayscale segmentation and residual net transfer learning, which reaches residual net level accuracy on image-based fire detection tasks with less computation. We train the model with a self-built image dataset, test it on a general fire image dataset, and evaluate it on a self-built fire video dataset. In numerous experiments, our method produced a training and testing accuracy of 98.26% and 99.90%, respectively, with corresponding losses of 0.036 and 0.04. The evaluation accuracy was 90% under the condition of a complex sample environment. Compared with other models, our method has advantages in accuracy, training loss, and cost time. Comprehensive comparisons show effectiveness and improved overall performance.