Hyperspectral LiDAR (HSL) has been widely discussed in recent years, which attracts increasing attention of the researchers in the field of electronic information technology. With the application of supercontinuum laser source, it is now possible to develop an HSL system, which can collect spectral and spatial information of targets simultaneously. Meanwhile, eye-safety and miniature HSL device with multiple spectral bands are given more priorities in on-site applications. In this paper, we tempt to investigate how to select spectral bands with a selection method. The proposed method consists of three steps: first, the variances among the classes based on hyperspectral feature parameters, termed inter-class variances, are calculated; second, the channels are sorted based on corresponding variances in descending order, and those with the two highest values are adopted as the initial input of classification; finally, the channels are selected successively from the rest of the sorted sequence until the classification accuracy reaches 100%. To test the performance of the proposed method, we collect 91/71-channel hyperspectral measurements of four different categories of materials with 5 nm spectral resolution using an acousto-optic tunable filter (AOTF) based HSL. Experimental results demonstrate that the proposed method could achieve higher classification accuracy than a random band selection method with different classifiers (naïve Bayes (NB) and support vector machine (SVM)) regardless of classification feature parameters (echo maximum and reflectance). To reach 100% accuracy, it demands 8–9 channels on average by echo maximum and 4–5 channels on average by reflectance based on NB classifier; these figures are 3–4 by echo maximum and 2–3 by reflectance with SVM classifier. The proposed method can complete classification task much faster than the random selection method. We further confirm the specific channels for the classification of different materials, and find that the optimal channels vary with different materials. The experimental results prove that the optimal band selection of HSL system for classification is reliable.