In this study, we investigated the feasibility of constructing a peaberry identification system using visible and fluorescence images to automate the peaberry sorting process in coffee beans. Coffee beans were irradiated with a white LED and a 360 nm ultraviolet LED, respectively, and photographed with a color camera. The RGB values of fluorescence images were converted into HSV (Hue, Saturation, Value) values, and the comparison of each feature value showed that there was a significant difference in the saturation of the fluorescence images. The Peaberry test data were discriminated against using a convolutional neural network trained on visible and fluorescence images as input, and the Peaberry discrimination rate was more than 95%. Furthermore, to verify the effectiveness of the system that combines visible and fluorescence images for classification, we randomly picked up a total of 56 beans, 28 of which were misclassified by visible and fluorescence images, and judged the type again using the sum of the output values of both classifiers using visible and fluorescence images, and found that 72%, or 41, were classified correctly. That suggests that the use of multiple image information contributes to improving the accuracy of the peaberry identification system.