It is a well-known fact that the quality of a seed highly impacts the germination of a rice seed. The age of the seed is one of the primary key points in assessing the seed quality. Therefore, this study aims to develop an AI-based machine-learning model to classify age-wise rice seeds. This study employs the SURF-BOF-based Cascaded-ANFIS algorithm for the implementation of the classifier. The proposed model performances were compared to the VGG16. Moreover, this research contributes a novel Japanese rice seed dataset to the scientific community. Furthermore, a 10-Fold cross-validation is performed to evaluate the robustness of the novel approach. The K-fold cross-validation's mean accuracy confirmed the proposed algorithm's higher robustness in the age-wise classification of rice seeds. Nevertheless, the results were evaluated using the confusion matrix and metrics such as precision, recall, and F1-Score. The Accuracy of Akitakomachi, Koshihikari, Yandao-8, and rice variety classification is 99%, 99%, 92%, and 97%. Analysis of the results determines the ability to classify rice by age and the general robustness of the algorithm.