Saffron (Crocus sativus L.) is the most expensive spice in the world, known for its unique aroma and coloring in the food industry. Hence, its high price is frequently adulterated. In the current study, a variety of soft computing methods, including classifiers (i.e., RBF, MLP, KNN, SVM, SOM, and LVQ), were employed to classify four samples of fake saffron (dyed citrus blossom, safflower, dyed fibers, and mixed stigma with stamens) and three samples of genuine saffron (dried by different methods). RGB and spectral images (near-infrared and red bands) were captured from prepared samples for analysis. The amount of crocin, safranal, and picrocrocin were measured chemically to compare the images’ analysis results. The comparison results of the classifiers indicated that KNN could classify RGB and NIR images of samples in the training phase with 100% accuracy. However, KNN’s accuracy for different samples in the test phase was between 71.31% and 88.10%. The RBF neural network achieved the highest accuracy in training, test, and total phases. The accuracy of 99.52% and 94.74% was obtained using the features extracted from RGB and spectral images, respectively. So, soft computing models are helpful tools for detecting and classifying fake and genuine saffron based on RGB and spectral images.