This paper proposed a novel image compression algorithm based on Gabor transform and fractal theory. First, texture features of image blocks were extracted based on Gabor transform; then cluster process were conducted for the R block and D block based on K-means algorithm, and in the process of block match, only those blocks belong to the same kind were taken into account, which can effectively reduce the encoding time. During the construction of the candidate blocks pool, two simplified approaches were introduced, including the simplification of the eight equidistant transforms and the outliers removed algorithm. Experimental results show that the proposed algorithm can both effectively speed up the encoding and ensure the image perfect visual effect, and has a good robustness.