An optical encryption (OE) scheme based on the spread spectrum ghost imaging (SSGI), named SSGI-OE, is proposed to obtain a high security with a smaller key. In the scheme, the randomly selected row number of a Hadamard matrix of order $N$ is used as the secure key, and shared with the authorized user, Bob, through a private channel. Each corresponding row vector of the order $N$ Hadamard matrix is then used as the direct sequence code to modulate a speckle pattern for the ghost imaging system, and an image is encrypted with the help of the SSGI. The measurement results from the bucket detector, named ciphertext, are then transmitted to Bob through a public channel. The illuminating speckle patterns are also shared with Bob by the public channel. With the correct secure key, Bob could reconstruct the image with the aid of the SSGI system, whereas, the unauthorized user, Eve, could not obtain any useful information of the encrypted image. The numerical simulations and experimental results show that the proposed scheme is feasible with a higher security and a smaller key. For the $32 \times 32$ pixels image, the number of bits sent from Alice to Bob by using SSGI-OE ($M=1024,N=2048$) scheme is only $0.0107$ times over the computational ghost imaging optical encryption(CGI-OE) scheme. When the eavesdropping ratio (ER) is less than 40%, the eavesdropper cannot acquire any information of the encrypted image. The extreme circumstance for the proposed SSGI-OE scheme is also discussed, where the eavesdropper begins to extract the information when ER is up to 15%.