For multi-color yarn-dyed fabrics which are cross-woven by yarns with different colors, the different colors cannot be directly measured by a traditional spectrophotometer because it can only obtain the average color of solid-color sample in the limited aperture. In this paper, a novel method for color segmentation and extraction for multi-color yarn-woven fabrics based on a Hyperspectral Imaging System (HIS) was proposed. First, the multi-color yarn-woven fabric images were acquired with the HIS. Then a space transformation based on Fréchet distance was used to transform the pre-processed hyperspectral fabric images into gray images, and then an improved watershed algorithm was used to segment the transformed gray images into different color regions. Finally, to solve the problems of over-segmentation with the improved watershed algorithm, an improved k-means clustering algorithm was adopted to merge the over-segmented color regions. The experimental results on four multi-color yarn-woven fabrics showed that the color segmentation accuracy of the proposed method outperformed the ordinary k-means, Fuzzy C-means (FCM), and Density peak cluster (DPC) algorithms on evaluation indexes of compactness (CP) and separation (SP), and the execution efficiency was improved by at least 55%. Furthermore, the color difference between the proposed method and the spectrophotometric measurements ranged from 0.60 to 0.88 CMC (2:1) (Color Measurement Committee) units, which almost satisfied the accuracy of color measurement.