Pedestrian detection plays a critical role in computer vision applications, but the performance of existing algorithms utilizing visible images is inadequate when confronted with poor lighting conditions. Conversely, infrared images deliver superior outcomes in low-light environments. Based on this issue, this study proposes the Double-EfficientDet-Improved (DEDI), a dual-stream model that integrates both visible and infrared imagery by enhancing the EfficientDet network structure. Firstly, the shuffle module is integrated into the backbone network to facilitate inter-channel information exchange within the feature layers. Secondly, by controlling the gradient path of the backbone network, the feature extraction ability is improved and the robustness is stronger. Lastly, a Multi-scale Fusion Module (MFM) is incorporated into the proposed model to harmonize information from both visible and infrared images. Experimental results demonstrate that this DEDI method achieves higher mean average precision (MAP) scores of 74.72\% on the KAIST dataset and 95.13\% on the LLVIP dataset, which is better than other object detection methods.