In this paper, a novel 3D image file encryption algorithm based on a new discrete chaotic system is presented. Chaotic characteristics of the novel chaotic system are analyzed by phase diagram, Lyapunov exponent, bifurcation diagram. Based on the analysis results, an encryption scheme is designed for 3D image file. The hash value which is calculated from the SHA-256 hash function is used to change the initial condition of the discrete system firstly. And then, the chaotic sequences are used to scramble and spread coordinate value for 3D image file by Arnold matrix and DNA diffusion algorithm. The security analysis shows that the proposed 3D image file encryption algorithm possesses higher security features to preserve the subject and resist conventional attack. The results of this paper give a different application scheme of the 3D image encryption algorithm.