Swarm intelligence algorithms have been widely used in the dimensional reduction of hyperspectral remote sensing imagery. The ant colony algorithm (ACA), the clone selection algorithm (CSA), particle swarm optimization (PSO), and the genetic algorithm (GA) are the most representative swarm intelligence algorithms and have often been used as subset generation procedures in the selection of optimal band subsets. However, studies on their comparative performance for band selection have been rare. For this paper, we employed ACA, CSA, PSO, GA, and a typical greedy algorithm (namely, sequential floating forward selection (SFFS)) as subset generation procedures and used the average Jeffreys–Matusita distance (JM) as the objective function. In this way, the band selection algorithm based on ACA (BS-ACA), band selection algorithm based on CSA (BS-CSA), band selection algorithm based on PSO (BS-PSO), band selection algorithm based on GA (BS-GA), and band selection algorithm based on SFFS (BS-SFFS) were tested and evaluated using two public datasets (the Indian Pines and Pavia University datasets). To evaluate the algorithms’ performance, the overall classification accuracy of maximum likelihood classifier and the average runtimes were calculated for band subsets of different sizes and were compared. The results show that the band subset selected by BS-PSO provides higher overall classification accuracy than the others and that its runtime is approximately equal to BS-GA’s, higher than those of BS-ACA, BS-CSA, and BS-SFFS. However, the premature characteristic of BS-ACA makes it unacceptable, and its average JM is lower than those of other algorithms. Furthermore, BS-PSO converged in 500 generations, whereas the other three swarm-intelligence based algorithms either ran into local optima or took more than 500 generations to converge. BS-PSO was thus proved to be an excellent band selection method for a hyperspectral image.