Bridges are prone to severe deterioration agents which promote their degradation over the course of their lifetime. Furthermore, maintenance budgets are being trimmed. This state of circumstances entails the development of a computer vision-based method for the condition assessment of bridge elements in an attempt to circumvent the drawbacks of visual inspection-based models. Scaling is progressive local flaking or loss in the surface portion of concrete that affects the functional and structural integrity of reinforced concrete bridges. As such, this research study proposes a self-adaptive three-tier method for the automated detection and assessment of scaling severity levels in reinforced concrete bridges. The first tier relies on the integration of cross entropy function and grey wolf optimization (GWO) algorithm for the segmentation of scaling pixels. The second tier is designated for the autonomous interpretation of scaling area. In this model, a hybrid feature extraction algorithm is proposed based on the fusion of singular value decomposition and discrete wavelet transform for the efficient and robust extraction of the most dominant features in scaling images. Then an integration of Elman neural network and GWO algorithm is proposed for the sake of improving the prediction accuracies of scaling area though optimization of both structure and parameters of Elman neural network. The third tier aims at establishing a unified scaling severity index to assess the extent of severities of scaling according to its area and depth. The developed method is validated through multi-layered comparative analysis that involved performance evaluation comparisons, statistical comparisons and box plots. Results demonstrated that the developed scaling detection model significantly outperformed a set of widely-utilized classical segmentation models achieving mean squared error, mean absolute error, peak signal to noise ratio and cross entropy of 0.175, 0.407, 55.754 and 26011.019, respectively. With regards to the developed scaling evaluation model, it accomplished remarkable better and more robust performance that other meta-heuristic-based Elman neural network models and conventional prediction models. In this context, it obtained mean absolute percentage error, root-mean squared error and mean absolute error 1.513%, 29.836 and 12.066, respectively, as per split validation. It is anticipated that the developed integrated computer vision-based method could serve as the basis of automated, reliable and cost-effective inspection platform of reinforced concrete bridges which can assist departments of transportation in taking effective preventive maintenance and rehabilitation actions.