Onychomycosis is the most common nail fungus disease in clinical practice worldwide, caused by the localization of various fungal agents, including dermatophytes, on the nail. The tests traditionally used for diagnosing onychomycosis are native examination, histopathological examination with periodic acid Schiff (PAS) staining, and nail culture. There is no gold standard method for diagnosing the disease, and the diagnosis process is time-consuming, costly, and quite laborious. Today, new technologies are needed to detect onychomycosis via AI-based ML to reduce the clinician and laboratory-induced error rate and increase diagnostic sensitivity and reliability. The present study aimed to design a decision support system to help the specialist doctor detect toenail fungus with artificial intelligence-based image processing techniques. The toenail images were taken by any camera initially from the individuals referred to the clinic. The image is divided into 12 RGB channels. Three hundred features were removed from each channel as 25 in the time domain. The best features were selected through feature selection algorithms in the next step to increase the performance and reduce the number of features, and models were created by algorithm classification. The average performance values of all proposed models, accuracy, sensitivity, and specificity, are 89.65, 0.9, and 0.89, respectively. The performance values of the most successful model-created accuracy, sensitivity, and specificity are 97.25, 0.96, and 0.98, respectively. Although the proposed method, according to the findings obtained in the study, has many advantages compared to the literature, it can be used as a decision support system for clinician diagnosis.