The transition to remote working due to the pandemic has accentuated the importance of clean indoor air, as people spend a significant portion of their time indoors. Amongst the various determinants of indoor air quality, outdoor pollution is a significant source. While conventional studies have certainly helped to quantify the long-term personal exposure to pollutants and assess their health impact, they have not paid special attention to the mechanism of transmission of pollutants between the two environments. Nevertheless, the quantification of infiltration is essential to determine the contribution of ambient pollutants in indoor air quality and its determinants. This study evaluates the transmission of outdoor pollutants into the indoor environment using 3D computational fluid dynamics modelling with a pollution dispersion model. Naturally ventilated buildings next to an urban canyon were modelled and simulated using Ansys Fluent and validated against wind tunnel results from the Concentration Data of Street Canyons database. The model consisted of two buildings of three storeys each, located on either side of a road. Two line-source pollutants were placed in the street, representing traffic emissions. Three internal rooms were selected and modelled on each floor and implemented with various ventilation strategies. Results indicate that for a canyon with an aspect ratio of 1, indoor spaces in upstream buildings are usually less polluted than downstream ones. Although within the canyon, pollution is 2–3 times higher near the upstream building. Cross ventilation can minimise or prevent infiltration of road-side pollutants into indoor spaces, while also assisting in the dispersion of ambient pollutants. The critical configuration, in terms of air quality, is single-sided ventilation from the canyon. This significantly increases indoor pollutant concentration regardless of the building location. The study reveals that multiple factors determine the indoor–outdoor links, and thorough indexing and understanding of the processes can help designers and urban planners in regulating urban configuration and geometries for improved indoor air quality. Future works should look at investigating the influence of indoor emissions and the effects of different seasons.