This study presents a novel computer vision based approach to automatically identify rutting appeared on asphalt pavement of road. The developed model is established base on a hybridization of image processing techniques and an advanced machine learning model with support of a metaheuristic optimization engine. Gabor filter and discrete cosine transform are employed to implement context computation for image data, accordingly generate initially extracted features of rutting and non-rutting. Least Squares Support Vector Classification (LSSVC) is then used to learn categorization of rutting and non-rutting based on the extracted features. The final LSSVC prediction model is constructed via a loop of optimization process which is controlled by a novel metaheuristic optimization algorithm, called forensic-based investigation (FBI), to attain optimal model's configuration with ultimate prediction accuracy. This study further utilized a dynamic feature selection (FS) method to integrate in the searching loop to appropriately remove redundant features that provide inconsistent information leading to the compromising of model performance. A dataset of 2000 image samples has been collected during field trip of pavement survey in Da Nang city to form and verify the newly developed model. The statistical results of 20 run times using k-fold cross validation method have demonstrated the hybrid model of FBI-LSSVC-FS to achieve the most desired rutting detection performance with classification accuracy rate, precision, recall, and F1 score of 98.9%, 0.994, 0.984 and 0.989, respectively. Hence, this paper contributes to the core body of knowledge a novel AI-based prediction model to assist transportation agencies in the task of periodic asphalt pavement survey.