Compared with integer order chaotic systems, fractional order chaotic systems can reflect natural phenomena more accurately, which are more suitable for chaotic cryptosystems. In order to explore the application of fractional order chaotic system in cryptography, a novel fractional order hyperchaotic system is constructed and implemented on DSP platform. More progressively, based on Adomian decomposition method, the dynamic behavior is studied by phase diagram, bifurcation diagram, Lyapunov exponent spectrum and spectral entropy (SE) complexity. It is found that each parameter and order have a large range of intervals that can keep the system in a hyperchaotic state. Therefore, the hyperchaotic sequences generated by the constructed fractional order hyperchaotic system have sufficient randomness and are well suited for applications in secure communications. In addition, a color image encryption scheme is designed based on the fractional order hyperchaotic system and DNA dynamic coding. Firstly, the improved Arnold algorithm is used to scramble the original image, then the column cyclic shift method is applied for secondary scrambling, and finally the pixel value is diffused by DNA sequence operation. The security analysis results indicate that the designed encryption algorithm can not only encrypt images effectively, but also has high security and can resist various common attacks.