Recently, many encryption algorithms based on fractional-order chaotic system have been proposed to solve the problem of image encryption. In this paper, we propose a novel color image compression-encryption algorithm based on fractional-order hyperchaotic system and DNA coding. In the data compression stage, the data of R, G, and B channels of the color image are converted to the frequency domain by two-dimensional discrete cosine transform (DCT), and then the amount of encrypted data is reduced by the quantization process. We design a data processing algorithm to ensure that the data after DCT is compatible with DNA coding data format. In the encryption stage, the processes of DNA encoding and decoding, DNA operation, and pixel scrambling are all controlled by the corresponding chaotic sequences, which are generated by the chaotic system. The original image is used to calculate the initial state of the chaotic system, which improves the performance of the algorithm against the chosen-plaintext attack significantly. Experimental results and security analysis illustrate that the proposed algorithm has excellent compression and security performance. It can not only reconstruct the original image well under the condition of low compression ratio, but also provide high security to resist various attacks. Besides, experimental results also indicate that the algorithm proposed can be applied to the fields of color image compression, encryption, and transmission. INDEX TERMS Color image encryption, DNA encoding, fractional-order hyperchaotic system, discrete cosine transform(DCT), image compression.