In this paper, we propose an optical voice encryption scheme based on speckle-illuminated Fourier ptychography (FP) and plaintext-related chaotic random phase mask (CRPM). In this proposed encryption scheme, the plaintext-related CRPMs are generated by chaotic Lozi map and secure hash algorithm (SHA-256). During the encryption process, the voice signal to be encrypted is first converted into a two-dimensional (2D) voice map. Then, with the help of CRPMs and speckle-illuminated FP, the voice map is encrypted into a series of noise-like low-resolution images. During the decryption process, the original voice signal can be recovered from the series of noise-like low-resolution images via Fourier ptychographic phase retrieval algorithm and the CRPMs. To the best of our knowledge, it is the first time to use the Fourier ptychography and chaotic random phase mask to implement the optical encryption of the voice signal. In addition, in this proposed encryption scheme, the chaotic parameters can replace the whole random phase masks as the secret keys, which makes the management and transmission of the secret keys become more convenient. Moreover, since the plaintext-related chaotic parameter keys can be updated dynamically, the security of the proposed encryption scheme can be further improved. The feasibility, security and robustness of the proposed encryption scheme are further analyzed by numerical simulations.