In this paper, a new method for generating multi-scroll chaotic attractors by introducing a new memristor model into an improved Chua’s system is presented. The dynamic characteristics of the system are analyzed, including equilibria and stability, bifurcation diagrams, Lyapunov exponents and phase diagrams. Strikingly, it can exhibit three-scroll, four-scroll and five-scroll hidden attractors by only changing one system parameter and different shapes of chaotic attractors coexist with the same parameter. Furthermore, the high randomness of the system is verified through 0–1 test and NIST test. By using the high randomness of the system, an image encryption algorithm (IEA) is designed to guarantee the secure and efficient transmission of digital images. This IEA uses whirlpool transformation to scramble pixel positions and DNA computation to diffuse pixel values. The simulation and performance analysis results indicate that the proposed IEA has high security and reliable encryption performance, which in turn confirms the availability of the new memristor-based Chua’s chaotic system.