Existing bridges are aging and deteriorating, raising concerns for public safety and the preservation of these valuable assets. Furthermore, the transportation networks that manage many bridges face budgetary constraints. This state of affairs necessitates the development of a computer vision-based method to alleviate shortcomings in visual inspection-based methods. In this context, the present study proposes a three-tier method for the automated detection and recognition of bridge defects. In the first tier, singular value decomposition ([Formula: see text]) is adopted to formulate the feature vector set through mapping the most dominant spatial domain features in images. The second tier encompasses a hybridization of the Elman neural network ([Formula: see text]) and the invasive weed optimization (I[Formula: see text]) algorithm to enhance the prediction performance of the ENN. This is accomplished by designing a variable optimization mechanism that aims at searching for the optimum exploration–exploitation trade-off in the neural network. The third tier involves validation through comparisons against a set of conventional machine-learning and deep-learning models capitalizing on performance prediction and statistical significance tests. A computerized platform was programmed in C#.net to facilitate implementation by the users. It was found that the method developed outperformed other prediction models achieving overall accuracy, F-measure, Kappa coefficient, balanced accuracy, Matthews’s correlation coefficient, and area under curve of 0.955, 0.955, 0.914, 0.965, 0.937, and 0.904, respectively as per cross validation. It is expected that the method developed can improve the decision-making process in bridge management systems.