The identification of sweet corn seed vitality is an essential criterion for selecting high-quality varieties. In this research, a combination of hyperspectral imaging technique and diverse deep learning algorithms has been utilized to identify different vitality grades of sweet corn seeds. First, the hyperspectral data of 496 seeds, including four viability-grade seeds, are extracted and preprocessed. Then, support vector machine (SVM) and extreme learning machine (ELM) are used to construct the classification models. Finally, the one-dimensional convolutional neural networks (1DCNN), one-dimensional long short-term memory (1DLSTM), the CNN combined with the LSTM (CNN-LSTM), and the proposed firefly algorithm (FA) optimized CNN-LSTM (FA-CNN-LSTM) are utilized to distinguish spectral images of sweet corn seeds viability grade. The findings from the experimental analysis indicate that the deep learning models exhibit a significant advantage over traditional machine learning approaches in the discrimination of seed vitality levels, boasting a classification accuracy exceeding 94.26% in test datasets and achieving an accuracy improvement of at least 3% compared to the best-performing machine learning model. Moreover, the performance of the FA-CNN-LSTM model proposed in this study demonstrated a slight superiority over the other three models. Besides, the FA-CNN-LSTM achieved a classification accuracy of 97.23%, representing a significant improvement of 2.97% compared to the lowest-performing CNN and a 1.49% enhancement over the CNN-LSTM. In summary, this study reveals the potential of integrating deep learning with hyperspectral imaging as a promising alternative for discriminating sweet corn seed vitality grade, showcasing its value in agricultural research and cultivar breeding.