Lesions of tea (Camellia sinensis) leaves are detrimental to the growth of tea crops. Their adverse effects include further disease of tea leaves and a direct reduction in yield and profit. Therefore, early detection and on‐site monitoring of tea leaf lesions are necessary for effective management to control infections and prevent further yield loss. In this study, 1,822 images of tea leaves with lesions caused by three diseases (brown blight, Colletotrichum camelliae; blister blight, Exobasidium vexans; and algal leaf spot, Cephaleuros virescens) and four pests (leaf miner, Tropicomyia theae; tea thrip, Scirtothrips dorsalis; tea leaf roller, Homona magnanima; and tea mosquito bug, Helopeltis fasciaticollis) were collected from northern and central Taiwan. A faster region‐based convolutional neural network (Faster R‐CNN) was then trained to detect the locations of the lesions on the leaves and to identify the causes of the lesions. The trained Faster R‐CNN detector achieved a precision of 77.5%, recall of 70.6%, an F1 score of 73.91%, and a mean average precision of 66.02%. An overall accuracy of 89.4% was obtained for identification of the seven classes of tea diseases and pests. The developed detector could assist tea farmers in identifying the causes of lesions in real time.