It is well known that, compared to low-dimension chaotic systems, three-dimensional chaotic systems have a wider parameter range, more complicated behaviour, and better unpredictability. This fact motivated us to introduce a novel image encryption method that employs a three-dimensional chaotic system. We proposed a novel three-dimensional conservative system that can exhibit chaotic behaviour involving hyperbolic functions. The dynamical behaviours of the proposed system are discovered by calculating Lyapunov exponents and bifurcation diagrams. Thereafter, we designed an image encryption method based on the proposed system and a 4×4 self-invertible matrix. A modified Diffie–Hellman key exchange protocol was utilised to generate the self-invertible key matrix Km employed in the diffusion stage. Our approach has three main stages. In the first stage, the proposed three-dimensional system utilises the original image to create three sequences, two of which are chosen for confusion and diffusion processes. The next stage involves confusing the image’s pixels by changing the positions of pixels using these sequences. In the third stage, the confused image is split into sub-blocks of size 4×4, and each block is encrypted by multiplying it with Km. Simulation findings demonstrated that the proposed image scheme has a high level of security and is resistant to statistical analysis, noise, and other attacks.