Deploying deep convolutional neural networks on agricultural devices with limited resources is challenging due to their large number of parameters. Existing lightweight networks can alleviate this problem but suffer from low performance. To this end, we propose a novel lightweight network named YOLOv8-RCAA (YOLOv8-RepVGG-CBAM-Anchorfree-ATSS), aiming to locate and detect tea leaf diseases with high accuracy and performance. Specifically, we employ RepVGG to replace CSPDarkNet63 to enhance feature extraction capability and inference efficiency. Then, we introduce CBAM attention to FPN and PAN in the neck layer to enhance the model perception of channel and spatial features. Additionally, an anchor-based detection head is replaced by an anchor-free head to further accelerate inference. Finally, we adopt the ATSS algorithm to adapt the allocating strategy of positive and negative samples during training to further enhance performance. Extensive experiments show that our model achieves precision, recall, F1 score, and mAP of 98.23%, 85.34%, 91.33%, and 98.14%, outperforming the traditional models by 4.22~6.61%, 2.89~4.65%, 3.48~5.52%, and 4.64~8.04%, respectively. Moreover, this model has a near-real-time inference speed, which provides technical support for deploying on agriculture devices. This study can reduce labor costs associated with the detection and prevention of tea leaf diseases. Additionally, it is expected to promote the integration of rapid disease detection into agricultural machinery in the future, thereby advancing the implementation of AI in agriculture.