The issue of information security in photonics environment has attracted more and more attention, especially when the secure communication of optical digital images has become a research hotspot. In this paper, we propose a hybrid encryption scheme for color images in the frequency and spatial domains based on double random transform and memristor hyperchaotic system. Firstly, we decompose the color image into RGB channels and then perform fast fourier transform (FFT) to transform the digital image from the spatial domain to the frequency domain. Secondly, two-phase masks are generated using the memristor hyperchaotic system, and the Fresnel diffraction optical transformation method is performed twice for encryption. Thirdly, the transformation from the frequency domain to the spatial domain is completed using inverse fast fourier transform (IFFT). Finally, the image is permuted and diffused using the chaotic sequences to obtain the final cipher image. Double random phase encryption expands the key space, while the combination of spatial and frequency domains improves the resistance to attacks. Based on cryptanalysis theory, we introduce a dynamic key associated with plain image, which can effectively resist plain attack. The experimental results show that the optical digital image encryption scheme has a large key space, excellent comprehensive performance, and can resist common attacks. In addition, we verify the hardware feasibility and ease of implementation of the proposed algorithm in an embedded optical communication network experimental platform. Therefore, our proposed scheme in this paper is a preferred optical digital image secure communication technology scheme with good application prospects.