Driver performances could be significantly impaired in adverse weather because of poor visibility and slippery roadways. Therefore, providing drivers with accurate weather information in real time is vital for safe driving. The state-of-practice of collecting roadway weather information is based on weather stations, which are expensive and cannot provide trajectory-level weather information. Therefore, the primary objective of this study was to develop an affordable detection system capable of providing trajectory-level weather information at the road surface level in real-time. This study utilized the Strategic Highway Research Program 2 Naturalistic Driving Study video data combined with a promising machine learning technique, called convolutional neural network (CNN), to develop a weather detection model with seven weather categories: clear, light rain, heavy rain, light snow, heavy snow, distant fog, and near fog. A novel CNN architecture, named RoadweatherNet, was carefully crafted to achieve the weather detection task. The evaluation results based on a test dataset revealed that RoadweatherNet can provide excellent performance in detecting weather conditions with an overall accuracy of 93%. The performance of RoadweatherNet was also compared with six pre-trained CNN models, namely, AlexNet, ResNet18, ResNet50, GoogLeNet, ShuffleNet, and SqueezeNet, which showed that RoadweatherNet can provide nearly identical performance with a significant reduction in training time. The proposed weather detection model is cost-efficient and requires less computational power; therefore, it can be made widely available mainly owing to the recent thriving of smartphone cameras and can be used to expand and update the current weather-based variable speed limit systems.