Polluted insulators seriously threaten the safe and stable operation of power grids, which attaches great significance to insulator contamination perception. Among the present methods, the non-contact approaches based on infrared images have gradually been widely used, as they are much more safe and are of low cost. However, the thermal effect of insulators is largely affected by meteorological conditions, which makes the infrared image-based methods less accurate. To solve the above problem, we take infrared image and meteorological parameters including humidity and temperature as input, and propose a feature fusion model to perceive insulator contamination in different weather conditions. Firstly, different feature extraction networks are used to perform feature extraction on the two types of data; secondly, the two features are concatenated to fuse together; thirdly, further feature extraction is performed and contamination is classified according to the pollution severity. Case studies show that the proposed method can better explore the relationship between humidity, temperature and pollution level of the insulators, thus can better separate the contamination grades and outperform the conventional infrared image based methods.