Brown blight, target spot, and tea coal diseases are three major leaf diseases of tea plants, and Apolygus lucorum is a major pest in tea plantations. The traditional symptom recognition of tea leaf diseases and insect pests is mainly through manual identification, which has some problems, such as low accuracy, low efficiency, strong subjectivity, and so on. Therefore, it is very necessary to find a method that could effectively identify tea plants diseases and pests. In this study, we proposed a recognition framework of tea leaf disease and insect pest symptoms based on Mask R-CNN, wavelet transform and F-RNet. First, Mask R-CNN model was used to segment disease spots and insect spots from tea leaves. Second, the two-dimensional discrete wavelet transform was used to enhance the features of the disease spots and insect spots images, so as to obtain the images with four frequencies. Finally, the images of four frequencies were simultaneously input into the four-channeled residual network (F-RNet) to identify symptoms of tea leaf diseases and insect pests. The results showed that Mask R-CNN model could detect 98.7% of DSIS, which ensure that almost disease spots and insect spots can be extracted from leaves. The accuracy of F-RNet model is 88%, which is higher than that of the other models (like SVM, AlexNet, VGG16 and ResNet18). Therefore, this experimental framework can accurately segment and identify diseases and insect spots of tea leaves, which not only of great significance for the accurate identification of tea plant diseases and insect pests, but also of great value for further using artificial intelligence to carry out the comprehensive control of tea plant diseases and insect pests.