Volume holographic imaging (VHI) is a promising biomedical imaging tool that can simultaneously provide multi-depth or multispectral information. When a VHI system is probed with a broadband source, the intensity spreads in the horizontal direction, causing degradation of the image contrast. We theoretically analyzed the reason of the horizontal intensity spread, and the analysis was validated by the simulation and experimental results of the broadband impulse response of the VHI system. We proposed a deconvolution method to reduce the horizontal intensity spread and increase the image contrast. Imaging experiments with three different objects, including bright field illuminated USAF test target and lung tissue specimen and fluorescent beads, were carried out to test the performance of the proposed method. The results demonstrated that the proposed method can significantly improve the horizontal contrast of the image acquire by broadband VHI system.