Fractional-order chaos has complex dynamic behavior characteristics, so its application in secure communication has attracted much attention. Compared with the design of fractional-order chaos-based cipher, there are fewer researches on security analysis. This paper conducts a comprehensive security analysis of a color image encryption algorithm using a fractional-order hyperchaotic system (CIEA-FOHS). Experimental simulation based on excellent numerical statistical results supported that CIEA-FOHS is cryptographically secure. Yet, from the perspective of cryptanalysis, this paper found that CIEA-FOHS can be broken by a chosen-plaintext attack method owing to its some inherent security defects. Firstly, the diffusion part can be eliminated by choosing some special images with all the same pixel values. Secondly, the permutation-only part can be deciphered by some chosen plain images and the corresponding cipher images. Finally, using the equivalent diffusion and permutation keys obtained in the previous two steps, the original plain image can be recovered from a target cipher image. Theoretical analysis and experimental simulations show that the attack method is both effective and efficient. To enhance the security, some suggestions for improvement are given. The reported results would help the designers of chaotic cryptography pay more attention to the gap of complex chaotic system and secure cryptosystem.

No abstract