Aiming at the problems of uneven light reflectivity on the spherical surface and high similarity between the stems/calyxes and scars that exist in the detection of surface defects in apples, this paper proposed a defect detection method based on image segmentation and stem/calyx recognition to realize the detection and recognition of surface defects in apples. Preliminary defect segmentation results were obtained by eliminating the interference of light reflection inhomogeneity through adaptive bilateral filtering-based single-scale Retinex (SSR) luminance correction and using adaptive gamma correction to enhance the Retinex reflective layer, and later segmenting the Retinex reflective layer by using a region-growing algorithm. The texture features of apple surface defects under different image processing methods were analyzed based on the gray level co-occurrence matrix, and a support vector machine was introduced for binary classification to differentiate between stems/calyxes and scars. Deploying the proposed defect detection method into the embedded device OpenMV4H7Plus, the accuracy of stem/calyx recognition reached 93.7%, and the accuracy of scar detection reached 94.2%. It has conclusively been shown that the proposed defect detection method can effectively detect apple surface defects in the presence of uneven light reflectivity and stem/calyx interference.