The double random phase encoding (DRPE) system plays a significant role in encrypted systems. However, it is a linear system that leads to security holes in encrypted systems. To tackle this issue, this paper proposes a novel optical image encryption scheme that combines a chaotic S-box, DRPE, and an improved Arnold transformation (IAT). In particular, the encryption scheme designs a chaotic S-box to substitute an image. The chaotic S-box has the characteristics of high nonlinearity and low differential uniformity and is then introduced to enhance the security of the DRPE system. Chaotic S-boxes are resistant to algebraic attacks. An IAT is used to scramble an image encoded by the DRPE system. Meanwhile, three chaotic sequences are obtained by a nonlinear chaotic map in the proposed encryption scheme. One of them is used for XOR operation, and the other two chaotic sequences are explored to generate two random masks in the DRPE system. Simulation results and performance analysis show that the proposed encryption scheme is efficient and secure.