The transmission of digital image in Internet is convenient, but at the same time the great risk of information leakage is existed. To meet the requirements of image content security and real-time communication, we propose an image encryption algorithm based on the Matryoshka transform and modular-inverse matrix. Firstly, a new Matryoshka transform is designed to scramble the original image; secondly, the global chaotic exclusive or operation is performed on the scrambled image, and the diffused result is converted into an image block set; thirdly, the chaotic sequence is used to generate a set of modular odd matrix, and the set of the modular-inverse matrix is calculated; finally, the matrix operation of modular multiplication is performed on the image block set and the modular-inverse matrix set to generate the encrypted image. Meanwhile, the sender passes the secret keys to the recipient using Diffie-Hellman over the elliptic curve. Experimental results and algorithm analyses show that the proposed algorithm has the excellent encryption effect and the strong ability to resist common attacks. Therefore, the proposed algorithm can be applied to the image transmission under the network platform with strong security, desirable encryption effect and high efficiency.