Printed fabrics usually have multiple colors and intricate patterns, which make it difficult to directly measure the colors of the printed fabrics with a traditional spectrophotometer. However, a hyperspectral imaging system (HIS) can measure multiple colors since it acquires the spectral reflectance of a continuous band at every point of the fabric. For multiple-color printed fabrics, color segmentation is also very important. In this paper, color measurement of printed fabrics using the HIS was implemented; an algorithm which combines the self-organizing map (SOM) algorithm and the density peaks clustering (DPC) algorithm was then proposed to automatically determine the number of colors on the printed fabric and accurately segment the color regions for measurement. Firstly, the SOM algorithm was used to identify the main clusters, the DPC algorithm with Silhouette Index was then used to identify the optimal number of colors and merge the clusters. Experimental results show that this algorithm not only automatically determines the optimal number of colors for printed fabric and achieves accurate color segmentation, but requires less time for execution.