In this paper, an image encryption algorithm based on the H-fractal and dynamic self-invertible matrix is proposed. The H-fractal diffusion encryption method is firstly used in this encryption algorithm. This method crosses the pixels at both ends of the H-fractal, and it can enrich the means of pixel diffusion. The encryption algorithm we propose uses the Lorenz hyperchaotic system to generate pseudorandom sequences for pixel location scrambling and self-invertible matrix construction to scramble and diffuse images. To link the cipher image with the original image, the initial values of the Lorenz hyperchaotic system are determined using the original image, and it can enhance the security of the encryption algorithm. The security analysis shows that this algorithm is easy to implement. It has a large key space and strong key sensitivity and can effectively resist plaintext attacks.