For the application of multiusers, the arrangement and distribution of the keys is a much concerning problem in a cryptosystem. In this paper, we propose an optical encryption scheme with multiple users based on computational ghost imaging (CGI) and orthogonal modulation. The CGI encrypts the secret image into an intensity vector rather than a complex-valued matrix. This will bring convenience for post-processing and transmission of the ciphertext. The orthogonal vectors are taken as the address codes to distinguish users and avoid cross-talk. Only the decryption key and the address code owned by an authorized user are matched, the secret image belonging to him/her could be extracted from the ciphertext. Therefore, there are two security levels in the encryption scheme. The feasibility and property are verified by numerical simulations.