Introducing memristor into the chaotic system can enrich the dynamic behaviors of the chaotic system. We propose a symbolic function memristor model and introduce it into a chaotic system to construct a fractional-order 5D memristor chaotic system. Through dynamic analysis of equilibrium point, Lyapunov exponents, phase diagram and bifurcation diagram, it is found that the system has abundant dynamic behaviors, for example, the change of equilibrium point type with parameters, transient chaos, offset-boosting and a special kind of extreme multistability. And with the change of parameters, the attractor state and shape will appear rich changes. Then the correctness of the system is verified by circuit simulation. The chaotic system is introduced into the process of image encryption, and an encryption system is constructed, which is composed of Zigzag scrambling, Hilbert curve scrambling, DNA encryption and GF257 diffusion algorithm. Finally, through a variety of security verification, the results show that the encryption system has good security and can resist many kinds of attacks effectively.