(1) Background: Artificial intelligence (AI) and machine learning (ML) techniques are being more widely employed in the field of wind engineering. Nevertheless, there is a scarcity of research on the comfort of pedestrians in terms of wind conditions with respect to building design, particularly in historic sites. (2) Objectives: This research aims to evaluate ML- and computational fluid dynamics (CFD)-based pedestrian wind comfort (PWC) analysis outputs using a novel method that relies on the sophisticated handling of image data. The goal is to propose a novel assessment method to enhance the efficiency of AI models over different urban scenarios. (3) Methodology: The stages include the analysis of climate data, CFD analysis with OpenFOAM, ML analysis using Autodesk Forma, and comparisons of the CFD and ML results using a novel image similarity assessment method based on the SSIM, MSE, and PSNR metrics. (4) Conclusions: This study effectively demonstrates the considerable potential of utilizing ML as a supplementary tool for evaluating PWC. It maintains a high degree of accuracy and precision, allowing for rapid and effective assessments. The methodology for precise comparison of two visual outputs in the absence of numerical data allows for more objective and pertinent comparisons, as it eliminates any potential distortions. (5) Recommendations: Additional research can explore the integration of ML models with climate data and different case studies, thus expanding the scope of wind comfort studies.