The confusing use of Polygonati Rhizoma (PR) and Polygonati Odorati Rhizoma (POR) poses an unpredictable threat to the health of consumers. Sensitive, nondestructive, rapid, and multicomponent techniques for their detection are sought after. In this study, a low-cost, short-wavelength (898–1668 nm), and handheld near-infrared (NIR) spectrometer combined with multivariate spectral evaluation methods was used to establish calibration models for identifying PR and POR. NIR spectra were treated with a standard normal variate (SNV) before performing chemometric approaches. Then principal component analysis (PCA) and partial least squares discriminant analysis (PLS-DA) were tested for calibration model development. The PCA results showed that spectral differences existed between the two herbs. However, the evaluation techniques could not separate them with the required accuracy. The PLS-DA calibration model, on the other hand, could separate the two herbs according to their spectral information with the prediction accuracy of >98.3%. Thus, it has been proven that a rapid, green, and low-cost method to support on-site and practical inspection through a handheld NIR instrument has been established to identify PR and POR and ensure the safety of the clinical medication.