Aiming at the existing encrypted speech retrieval schemes that do not support multi-user multi-keyword retrieval, low retrieval efficiency and accuracy, lack of verifiability of encrypted speech retrieval results, and user fairness, a multi-user and multi-keyword verifiable encrypted speech retrieval method based on blockchain was proposed. Firstly, use the advanced encryption standard-128 (AES-128) to encrypt the speech and upload it to the cloud server. Secondly, the Mel-frequency cepstral coefficient (MFCC) and Filter bank (Fbank) speech features are extracted and fused into new MFCC-Fbank features, which are input into the designed CNN-BiGRU model for training. These features are used to extract more robust deep features to construct hash codes and are used as searchable encrypted keywords. Finally, use smart contracts to store encrypted speech hash codes and corresponding index values and manage multi-user keys through trusted institutions. When retrieval, smart contracts are used as trusted third parties for retrieval work, and the designed adaptive homomorphic Message Authentication Code (MAC) algorithm is used to verify the correctness of the retrieval results, ensuring fairness in verification and data privacy. The experimental results show that the proposed method effectively prevents user privacy leakage and improves encrypted speech retrieval accuracy and data security under multi-user and multi-keyword conditions.