Benefit from the high resolution, penetrating and all-weather advantages of millimeterwave (MMW) imaging, MMW imaging plays an important role in remote sensing, security inspection, navigation, etc. Among the MMW imaging systems, synthetic aperture imaging radiometer (SAIR) utilizes aperture synthetic technology to achieve higher imaging resolution, but the perception information is insufficient, resulting in poor image quality. To improve the image quality of passive SAIR MMW images effectively, we propose a novel multichannel depth convolutional neural network (MDCNN) in this paper. Aiming at the characteristics of original MMW images with more noise in low-frequency information and fewer features in high-frequency information, wavelet transform is incorporated into the MDCNN to obtain the high-and low-frequency components first. Then, dense residual block and skip connection technology are adopted to denoise and enhance target information in the four independent channels respectively. Finally, high-quality MMW images are synthesized by inverse wavelet transform. The simulation results show that the reconstructed images of MDCNN have better image quality (such as image contour and texture details) than other deep learning-based methods.