Conventional crop fertilization classification and fertilization decision-making methods are difficult to operate efficiently and accurately. In order to improve the efficiency of greenhouse tomato fertilization and fertilization decision-making, this study is based on more than 1,800 greenhouse tomato leaf pictures, using data enhancement, migration learning, gradient descent, the regularization method optimizes the Convolutional Neural Network (CNN), and builds a deep learning model for tomato leaf fertility classification. The full convolutional neural network (FCN) is optimized using data enhancement and hyperparameter optimization methods, and a deep learning model for tomato leaf region segmentation and region information extraction is constructed. In this paper, the leaf image information collected by the on-site camera is converted into improved color space information, combined with the tomato leaf nitrogen deficiency-color model, to realize the amount of decision of the tomato nitrogen fertilizer at the irrigation site. Research shows that the accuracy of the CNN leaf fertility recognition model training set is close to 100%, the accuracy of the validation set can reach 95%, and the accuracy of the test set is 91%; the average intersection ratio from the test set data of the FCN leaf segmentation model is 0.91 and the average pixel accuracy is 0.94.