Recognizing and identifying tea plant (Camellia sinensis) cultivar plays a significant role in tea planting and germplasm resource management, particularly for oolong tea. There is a wide range of high-quality oolong tea with diverse varieties of tea plants that are suitable for oolong tea production. The conventional method for identifying and confirming tea cultivars involves visual assessment. Machine learning and computer vision-based automatic classification methods offer efficient and non-invasive alternatives for rapid categorization. Despite advancements in technology, the identification and classification of tea cultivars still pose a complex challenge. This paper utilized machine learning approaches for classifying 18 oolong tea cultivars based on 27 multispectral characteristics. Then the SVM classification model was executed using three optimization algorithms, namely genetic algorithm (GA), particle swarm optimization (PSO), and grey wolf optimizer (GWO). The results revealed that the SVM model optimized by GWO achieved the best performance, with an average discrimination rate of 99.91%, 93.30% and 92.63% for the training set, test set and validation set, respectively. In addition, based on the multispectral information (h, s, r, b, L, Asm, Var, Hom, Dis, σ, S, G, RVI, DVI, VOG), the germination period of oolong tea cultivars can be completely evaluated by Fisher discriminant analysis. The study indicated that the practical protection of tea plants through automated and precise classification of oolong tea cultivars and germination periods is feasible by utilizing multispectral imaging system.