Pothole detection plays a crucial role in preventing road accidents and is effective in establishing road maintenance and safety. Although various pothole detection models are designed to accurately identify the pothole based on road images, they face issues in accuracy and hyperparameter tuning. The presented research work concentrates on developing a novel optimized deep learning model for the accurate prediction of potholes on the road infrastructure using the recurrent neural network (RNN) and grey wolf optimization (GWO). Initially, the road images are collected and pre-processed. The pre-processing includes the removal of noises, image resizing, etc., to improve the image quality. Further, texture-based feature extraction was employed to extract the most relevant features from the pre-processed image. Then, the RNN architecture was trained using the extracted features to learn the interconnections between the image features and pothole detection. In addition, the GWO fitness solution was integrated into the classification module to tune and optimize the RNN hyperparameters, which increases the detection performances such as accuracy, and reduces the loss function. Finally, the presented model was evaluated with the publically available road image detection database and the outcomes are determined. The performance assessment demonstrates that the designed model attained greater accuracy of 98.76%, and a loss function of 0.06. Furthermore, a comparative assessment was performed with existing methods to evaluate the effectiveness of the proposed model.