Rose tea is a type of flower tea in China’s reprocessed tea category, which is divided into seven grades, including super flower, primary flower, flower bud, flower heart, yellow flower, scattered flower, and waste flower. Grading rose tea into distinct quality levels is a practice that is essential to boosting their competitive advantage. Manual grading is inefficient. We provide a lightweight model to advance rose tea grading automation. Firstly, four kinds of attention mechanisms were introduced into the backbone and compared. According to the experimental results, the Convolutional Block Attention Module (CBAM) was chosen in the end due to its ultimate capacity to enhance the overall detection performance of the model. Second, the lightweight module C2fGhost was utilized to change the original C2f module in the neck to lighten the network while maintaining detection performance. Finally, we used the SIoU loss in place of the CIoU loss to improve the boundary regression performance of the model. The results showed that the mAP, precision (P), recall (R), FPS, GFLOPs, and Params values of the proposed model were 86.16%, 89.77%, 83.01%, 166.58, 7.978, and 2.746 M, respectively. Compared with the original model, the mAP, P, and R values increased by 0.67%, 0.73%, and 0.64%, the GFLOPs and Params decreased by 0.88 and 0.411 M, respectively, and the speed was comparable. The model proposed in this study also performed better than other advanced detection models. It provides theoretical research and technical support for the intelligent grading of roses.