The bioactive components of chrysanthemum tea are an essential indicator in evaluating its nutritive and commercial values. Combining hyperspectral imaging (HSI) with key wavelength selection and pattern recognition methods, this study developed a novel approach to estimating the content of bioactive components in chrysanthemums, including the total flavonoids (TFs) and chlorogenic acids (TCAs). To determine the informative wavelengths of hyperspectral images, we introduced a variable similarity regularization term into particle swarm optimization (denoted as VSPSO), which can focus on improving the combinatorial performance of key wavelengths and filtering out the features with higher collinearity simultaneously. Moreover, considering the underlying relevance of the phytochemical content and the exterior morphology characteristics, the spatial image features were also extracted. Finally, an ensemble learning model, LightGBM, was established to estimate the TF and TCA contents using the fused features. Experimental results indicated that the proposed VSPSO achieved a superior accuracy, with R2 scores of 0.9280 and 0.8882 for TF and TCA prediction. Furthermore, after the involvement of spatial image information, the fused spectral–spatial features achieved the optimal model accuracy on LightGBM. The R2 scores reached 0.9541 and 0.9137, increasing by 0.0308–0.1404 and 0.0181–0.1066 in comparison with classical wavelength-related methods and models. Overall, our research provides a novel method for estimating the bioactive components in chrysanthemum tea accurately and efficiently. These discoveries revealed the potential effectiveness for constructing feature fusion in HSI-based practical applications, such as nutritive value evaluation and heavy metal pollution detection, which will also facilitate the development of quality detection in the food industry.