Mango is a popular fruit for local consumption and export commodity. Currently, Indonesian mango export at 37.8 M accounted for 0.115% of world consumption. Pest and disease are the common enemies of mango that degrade the quality of mango yield. Specialized treatment in export destinations such as gamma-ray in Australia, or hot water treatment in Korea, demands pest-free and high-quality products. Artificial intelligence helps to improve mango pest and disease control. This paper compares the deep learning model on mango fruit pests and disease recognition. This research compares Visual Geometry Group 16 (VGG16), residual neural network 50 (ResNet50), InceptionResNet-V2, Inception-V3, and DenseNet architectures to identify pests and diseases on mango fruit. We implement transfer learning, adopt all pre-trained weight parameters from all those architectures, and replace the final layer to adjust the output. All the architectures are re-train and validated using our dataset. The tropical mango dataset is collected and labeled by a subject matter expert. The VGG16 model achieves the top validation and testing accuracy at 89% and 90%, respectively. VGG16 is the shallowest model, with 16 layers; therefore, the model was the smallest size. The testing time is superior to the rest of the experiment at 2 seconds for 130 testing images.