The safety and serviceability of civil infrastructures have to be ensured either as part of a periodic inspection program or immediately following a given hazardous event. The use of digital imaging techniques to identify the deformed or deteriorated surfaces of structures is a substantial area of research and aims to investigate a number of unknown parameters, including damage quantification and condition rating. This manuscript illustrates the integration of previously developed fuzzy logic-based decision-making tools with the currently developed image processing algorithm to quantify the damage for the condition rating of civil infrastructures. The proposed integrated framework exploits visual specifics of different elements of the infrastructure to perform automated evaluation of structural anomalies such as cracks and surface degradation. Two different image segmentation tools, (1) bottom hat transform and (2) hue, saturation, color (HSV) thresholding, are applied to identify the surface defects. The developed image processing software is used with the fuzzy set framework proposed in the previous research to gauge the damage indices due to various deterioration types like corrosion, alkali aggregate reaction, freeze-thaw attack, sulfate attack, acid attack or loading, fatigue, shrinkage, and honeycombing. Case studies of a long-span bridge and a warehouse building are illustrated for concept validation. The refined comprehensive method is presented as a graphical user interface (GUI) to facilitate the real-time condition assessment of civil infrastructures.