Images of different modalities play important roles in the fields of military, navigation, and target detection, including visible and infrared images. Existing fusion methods can reach relatively good fusion effects, but often make the processing speed slow. To achieve the purpose of faster fusion, this paper proposes a fast fusion of visible and infrared images (FFVI) based on Fourier transform and difference minimization. First, both visible and infrared images are transformed using Fourier transform, and the contour information of infrared image is extracted out at the same time. Then the two transformed images are added together to obtain a composite image, the inverse Fourier transform and the grayscale normalization are performed on the composite transformation image, which will generate an image that reflects the target features. Finally, the resulting image is reconstructed by fusing visible and the generated images according to the principle of "difference minimization". The experimental results show that the novel method FFVI proposed in this paper can attain remarkable fusion effects in terms of the subjective qualitative and the objective quantitative analysis. In addition, this method outperforms several representative image fusion algorithms in the computational speed, which will be beneficial to practical application.