Fabric shape retention is one of the most important attributes of fabrics that can influence the quality of the end use product. In this paper, we present a computer vision-based method to analyze the sequential images, which records the dynamic change of a deformed fabric, to model the recovery process, and extract the features of the recovery curve to characterize the shape retention after the deformation. Image processing and the perceptual hash algorithm were used to convert the measurements of a fabric shape variable at different times into Hamming distance points. The recovery function of the fabric shape was formed by fitting the Hamming distance points with exponential function, and three new shape retention indexes, that is, the average slope, the abscissa of the inflation point, and the radius of curvature at the inflation point, were defined based on the recovery function. The experiment showed that the shape retention of 12 fabric samples after deformation could be effectively distinguished by the new indexes. This paper also discussed the relationships between the new indexes and the transitional measurements indicating the fabric shape retention.