Based on some analyses of existing chaotic image encryption frameworks and a new designed 3D-ILM chaotic map, an asymmetric image encryption algorithm using public-key RSA is presented in this paper. In the first stage, a new 3D-ILM chaotic map is proposed to enhance the chaotic behavior considering analysis of time sequence, Lyapunov exponent and Shannon entropy. In the second stage, combined with the public key RSA algorithm, a new key acquisition mathematical model (MKA) is constructed to obtain the initial keys for the 3D-ILM. Consequently, the keystream can be produced depending on the plain image for a higher security. Moreover, a novel process model (NPM) for the input of the 3D-ILM is built, which is built to improve the distribution uniformity of the chaotic sequence. In the third stage, to encrypt the plain image, a pre-process by XOR operation with a random matrix is applied. Then, the pre-processed image is performed by a permutation for rows, a downward modulo function for adjacent pixels, a permutation for columns, a forward direction XOR addition-modulo diffusion, and a backward direction XOR addition-modulo diffusion to achieve the final cipher image. Moreover, experiments show that the the proposed algorithm has a better performance. Especially, the number of pixels change rate (NPCR) is near to ideal case 99.6094%, with the unified average changing intensity (UACI) near to 33.4634% and the information entropy (IE) near to 8.