Most of the image encryption schemes based on chaos have so far employed symmetric key cryptography, which leads to a situation where the key cannot be transmitted in public channels, thus limiting their extended application. Based on the elliptic curve cryptography (ECC), we proposed a public key image encryption method where the hash value derived from the plain image was encrypted by ECC. Furthermore, during image permutation, a novel algorithm based on different-sized block was proposed. The plain image was firstly divided into five planes according to the amount of information contained in different bits: the combination of the low 4 bits, and other four planes of high 4 bits respectively. Second, for different planes, the corresponding method of block partition was followed by the rule that the higher the bit plane, the smaller the size of the partitioned block as a basic unit for permutation. In the diffusion phase, the used hyperchaotic sequences in permutation were applied to improve the efficiency. Lots of experimental simulations and cryptanalyses were implemented in which the NPCR and UACI are 99.6124% and 33.4600% respectively, which all suggested that it can effectively resist statistical analysis attacks and chosen plaintext attacks.