Dried tangerine peel is a Chinese medicine with high medicinal value. The storage age is an important indicator of its medicinal value, so it is very significant to accurately identify the storage age of dried tangerine peel. Traditional physical and chemical analysis methods can be used to achieve this goal, but these methods are limited by their operability and convenience. Near infrared (NIR) spectroscopy and machine learning have excellent performance in the rapid detection of food and pharmaceutical samples. This study investigated the novel application of integrating a hand-held NIR spectrometer combined with machine learning to rapidly and accurately identify the storage age of Xinhui dried tangerine peel. Savitzky–Golay convolution smoothing, standard normal variate (SNV), first derivative, and second derivative pretreatments were employed to preprocess spectral data. Principal component analysis (PCA) was used to reduce the spectral data dimensions and obtain the characteristic spectral variables of each sample. Support vector machine (SVM) and k-nearest neighbor were applied to establish the qualitative discriminant models. The SNV-PCA-SVM model discriminant accuracy was 99.60% in the validation set and was 96.50% in the test set, showing excellent generalization performance. The results indicated that the method of using a hand-held NIR spectrometer combined with machine learning could be applied to rapidly identify the storage age of Xinhui dried tangerine peel. This is a promising and economical hand-held NIR spectroscopic method for assuring the dried tangerine peel age on-site.