In order to improve the problems of poor accuracy and low efficiency in tobacco leaves disease recognition and diagnosis and avoid the misjudgment in tobacco disease recognition, a disease recognition and spot segmentation method based on the improved ORB algorithm was proposed. The improved FAST14-24 algorithm was used to preliminarily extract corners. It overcame the deficiency of the sensitivity of the traditional ORB corner detection algorithm to image edges. During the experiment, 28 parameters were obtained through the extraction of color features, morphological features, texture, and other features of tobacco disease spots. Through the experimental comparisons, it was found that the fitness of the improved ORB algorithm was 96.68 and the cross-checking rate was 93.21%. The validation and recognition rate for samples was 96%. The identification rate of tobacco brown spot disease and frog eye disease was 92%, and the identification rate of 6 categories in different periods was over 96%. The experimental results verified the effects of the disease identification fully.