It is now well established that biological pollution is a major cause of the degradation of indoor air quality. It has been shown that microbial communities from the outdoors may significantly impact the communities detected indoors. One can reasonably assume that the fungal contamination of the surfaces of building materials and their release into indoor air may also significantly impact indoor air quality. Fungi are well known as common contaminants of the indoor environment with the ability to grow on many types of building materials and to subsequently release biological particles into the indoor air. The aerosolization of allergenic compounds or mycotoxins borne by fungal particles or vehiculated by dust may have a direct impact on the occupant’s health. However, to date, very few studies have investigated such an impact. The present paper reviewed the available data on indoor fungal contamination in different types of buildings with the aim of highlighting the direct connections between the growth on indoor building materials and the degradation of indoor air quality through the aerosolization of mycotoxins. Some studies showed that average airborne fungal spore concentrations were higher in buildings where mould was a contaminant than in normal buildings and that there was a strong association between fungal contamination and health problems for occupants. In addition, the most frequent fungal species on surfaces are also those most commonly identified in indoor air, regardless the geographical location in Europe or the USA. Some fungal species contaminating the indoors may be dangerous for human health as they produce mycotoxins. These contaminants, when aerosolized with fungal particles, can be inhaled and may endanger human health. However, it appears that more work is needed to characterize the direct impact of surface contamination on the airborne fungal particle concentration. In addition, fungal species growing in buildings and their known mycotoxins are different from those contaminating foods. This is why further in situ studies to identify fungal contaminants at the species level and to quantify their average concentration on both surfaces and in the air are needed to be better predict health risks due to mycotoxin aerosolization.