Unfortunately, accidents caused by bad weather have regularly made headlines throughout history. Some of the more catastrophic events to recently make news include a plane crash, ship collision, railway derailment, and several vehicle accidents. The public’s attention has been directed to the severe issue of safety and security under extreme weather conditions, and many studies have been conducted to highlight the susceptibility of transportation services to environmental factors. An automated method of determining the weather’s state has gained importance with the development of new technologies and the rise of a new industry: intelligent transportation. Humans are well-suited for determining the temperature from a single photograph. Nevertheless, this is a more challenging problem for a fully autonomous system. The objective of this research is developing a good weather classifier that uses only a single image as input. To resolve quality-of-life challenges, we propose a modified deep-learning method to classify the weather condition. The proposed model is based on the Yolov5 model, which has been hyperparameter tuned with the Learning-without-Forgetting (LwF) approach. We took 1499 images from the Roboflow data repository and divided them into training, validation, and testing sets (70%, 20%, and 10%, respectively). The proposed model has gained 99.19% accuracy. The results demonstrated that the proposed model gained a much higher accuracy level in comparison with existing approaches. In the future, this proposed model may be implemented in real-time.