To address the shortcomings of weak confusion and high time complexity of the existing permutation algorithms, including the traditional Josephus ring permutation (TJRP), an improved Josephus ring-based permutation (IJRBP) algorithm is developed. The proposed IJRBP replaces the remove operation used in TJRP with the position exchange operation and employs random permutation steps instead of fixed steps, which can offer a better scrambling effect and a higher permutation efficiency, compared with various scrambling methods. Then, a new encryption algorithm based on the IJRBP and chaotic system is developed. In our scheme, the plaintext feature parameter, which is related to the plaintext and a random sequence generated by a chaotic system, is used as the shift step of the circular shift operation to generate the diffusion matrix, which means that a minor change in the source image will generate a totally different encrypted image. Such a strategy strikes a balance between plaintext sensitivity and ciphertext sensitivity to obtain the ability to resist chosen-plaintext attacks (CPAs) and the high robustness of resisting noise attacks and data loss. Simulation results demonstrate that the proposed image cryptosystem has the advantages of great encryption efficiency and the ability to resist various common attacks.