Xinhui tangerine peel has valuable medicinal value. The longer it is stored in an appropriate environment, the higher its flavonoid content, resulting in increased medicinal value. In order to correctly identify the age of the tangerine peel, previous studies have mostly used manual identification or physical and chemical analysis, which is a tedious and costly process. This work investigates the automatic age recognition of the tangerine peel based on deep learning and attention mechanisms. We proposed an effective ConvNeXt fusion attention module (CNFA), which consists of three parts, a ConvNeXt block for extracting low-level features’ information and aggregating hierarchical features, a channel squeeze-and-excitation (cSE) block and a spatial squeeze-and-excitation (sSE) block for generating sufficient high-level feature information from both channel and spatial dimensions. To analyze the features of tangerine peel in different ages and evaluate the performance of CNFA module, we conducted comparative experiments using the CNFA-integrated network on the Xinhui tangerine peel dataset. The proposed algorithm is compared with related models of the proposed structure and other attention mechanisms. The experimental results showed that the proposed algorithm had an accuracy of 97.17%, precision of 96.18%, recall of 96.09%, and F1 score of 96.13% for age recognition of the tangerine peel, providing a visual solution for the intelligent development of the tangerine peel industry.