Abstract. In this paper, a novel fusion algorithm is proposed based on visual brightness and contrast response function, and it can better retain the difference features between infrared and polarization images to improve the visual effect of fused image. Firstly, the source images are enhanced by fractional differentia operator to improve the edge, texture, and brightness. Second, the low frequency and high frequency features of enhanced infrared polarization and intensity image are described by local mean and local skewness. Thirdly, the visual brightness and contrast response function are constructed by using local mean and local skewness based on human visual model. Next, the source images are fused by using the visual brightness and contrast response function, and then two kinds of fused images that are respectively are the low frequency feature fused image and detail feature fused image are gotten. Finally, the fused image is gotten by superposition of the low frequency feature fused image and detail feature fused image. The experiment shows that the result obtained by proposed algorithm performs better in both subjective and objective qualities.