The current common color image encryption algorithms applying “scrambling-diffusion” have some problems, such as the small key space, the cumbersome encryption process, and the security vulnerability. Aiming at these problems, this paper proposes a new color image encryption algorithm based on the hyperchaotic system and applying “transforming-scrambling-diffusion” model. Before scrambling, in accordance with the plaintext itself attributes, the number of iterations was calculated, all the pixel values of color image were transformed into gray code iteratively, and then the chaotic sequence was generated from the four-dimensional hyperchaotic system. Pixel matrix after gray code transformation was converted to one-dimensional matrix. The chaotic sequence was sorted and the one-dimensional matrix was changed positions correspondingly to complete the whole domain scrambling. And then, bit-operation was executed for image diffusion. The ciphertext can be obtained by matrix transformation. The key sensitivity, histogram, information entropy, correlation, and other evaluation indexes were calculated and analyzed through the simulation experiment. Compared with other algorithms, it can be proved that the encryption algorithm has the strong antiattack ability.