Cultivar identification is a basic task in oil tea (Camellia oleifera C.Abel) breeding, quality analysis, and an adjustment in the industrial structure. However, because the differences in texture, shape, and color under different cultivars of oil tea are usually inconspicuous and subtle, the identification of oil tea cultivars can be a significant challenge. The main goal of this study is to propose an automatic and accurate method for identifying oil tea cultivars. In this study, a new deep learning model is built, called EfficientNet-B4-CBAM, to identify oil tea cultivars. First, 4725 images containing four cultivars were collected to build an oil tea cultivar identification dataset. EfficientNet-B4 was selected as the basic model of oil tea cultivar identification, and the Convolutional Block Attention Module (CBAM) was integrated into EfficientNet-B4 to build EfficientNet-B4-CBAM, thereby improving the focusing ability of the fruit areas and the information expression capability of the fruit areas. Finally, the cultivar identification capability of EfficientNet-B4-CBAM was tested on the testing dataset and compared with InceptionV3, VGG16, ResNet50, EfficientNet-B4, and EfficientNet-B4-SE. The experiment results showed that the EfficientNet-B4-CBAM model achieves an overall accuracy of 97.02% and a kappa coefficient of 0.96, which is higher than that of other methods used in comparative experiments. In addition, gradient-weighted class activation mapping network visualization also showed that EfficientNet-B4-CBAM can pay more attention to the fruit areas that play a key role in cultivar identification. This study provides new effective strategies and a theoretical basis for the application of deep learning technology in the identification of oil tea cultivars and provides technical support for the automatic identification and non-destructive testing of oil tea cultivars.