The defective beans of coffee are categorized into black beans, fermented beans, moldy beans, insect damaged beans, parchment beans, and broken beans, and insect damaged beans are the most frequently seen type. In the past, coffee beans were manually screened and eye strain would induce misrecognition. This paper used a push-broom visible-near infrared (VIS-NIR) hyperspectral sensor to obtain the images of coffee beans, and further developed a hyperspectral insect damage detection algorithm (HIDDA), which can automatically detect insect damaged beans using only a few bands and one spectral signature. First, by taking advantage of the constrained energy minimization (CEM) developed band selection methods, constrained energy minimization-constrained band dependence minimization (CEM-BDM), minimum variance band prioritization (MinV-BP), maximal variance-based bp (MaxV-BP), sequential forward CTBS (SF-CTBS), sequential backward CTBS (SB-CTBS), and principal component analysis (PCA) were used to select the bands, and then two classifier methods were further proposed. One combined CEM with support vector machine (SVM) for classification, while the other used convolutional neural networks (CNN) and deep learning for classification where six band selection methods were then analyzed. The experiments collected 1139 beans and 20 images, and the results demonstrated that only three bands are really need to achieve 95% of accuracy and 90% of kappa coefficient. These findings show that 850–950 nm is an important wavelength range for accurately identifying insect damaged beans, and HIDDA can indeed detect insect damaged beans with only one spectral signature, which will provide an advantage in the process of practical application and commercialization in the future.