As one of the most popular edible fungi in the market, the quality of Agaricus bisporus will determine its sales volume. Therefore, to achieve rapid and nondestructive testing of the quality of Agaricus bisporus, this study first built a portable spectrum acquisition device for Agaricus bisporus. The Ocean Spectromeper was used to calibrate the spectral data of the device, and the linear regression analysis method was combined to analyze the two. The results showed that the Pearson correlation coefficient of significance between the two was 0.98. Then, the spectral data of Agaricus bisporus were collected, the spectral characteristic wavelength of Agaricus bisporus was extracted by the SPA and PCA algorithms, and the moisture content and whiteness prediction models based on a BP neural network and PLSR, respectively, were built. The parameters of the BP neural network model were optimized by SSA. The R2 values for the final moisture content and the predicted whiteness were 0.95 and 0.99, and the RMSE values were 5.04% and 0.60, respectively. The results show that the portable spectral acquisition and analysis device can be used for the accurate and rapid quality detection of Agaricus bisporus.