Reconstructing images from speckle patterns using deep learning methods is emerging as an effective alternative to traditional approaches. To overcome the potential multiple diffuser distortions occurring between the emission and the detection of the optical path, we establish a 4-f imaging system incorporating dual diffusers, one positioned in front of the image plane and the other after the object plane, to capture plenty of scattered object images. To effectively reconstruct from the experimentally acquired speckle patterns, we add the Triple Attention Module into the UNeXt convolutional network (TAM-UNeXt) and concurrently preprocess the autocorrelation spectrum of the patterns inspired by the angular memory effect theory. We compare the recovery results of the TAM-UNeXt under various conditions, including different grit sizes, numbers, and positions of the diffusers, as well as several optical lens setups, to verify its adaptability under diverse double diffuser conditions.