We develop a novel method to encrypt a color image by combining enhanced optical interference with phase retrieval algorithm and linear phase color-blend (LPCB). In this method, the original color image is encoded into three phase only masks (POMs) with different Fresnel diffraction distances and one fixed random phase mask (RPM). For the decryption in an optical interference system, the three individual POMs are placed in the red, green and blue channels, respectively, and the RPM is placed in the public channel. Only if the right POMs and RPM are fixed in their positions properly, one could achieve the expected information. Compared with traditional interference based color image method, our method eliminates the silhouette problem fully. LPCB is also introduced to modulate the phase value of each pixel of the three POMs to further enhance the security. Numerical simulation results are presented to illustrate the feasibility and effectiveness of the proposed scheme.