Objective We aimed to assess the relationship between major air pollutants and the natural history and mortality of idiopathic pulmonary fibrosis (IPF). Methods We conducted a retrospective cohort study from 2013 to 2019 among 52 patients with IPF from the pneumology department of a tertiary hospital. According to their geocoded residential address, each patient was assigned a mean concentration of carbon monoxide (CO), nitrogen dioxide, particulate matter 2.5 and 10, ozone, and sulfur dioxide, as measured at a single surveillance station in central Madrid, Spain. We analyzed forced vital capacity (FVC), CO diffusing capacity, 6-minute walking test, degree of dyspnea, radiologic pattern, and signs of pulmonary hypertension in all patients. Results Patients’ mean age was 66 ± 10 years, and 79% were men. The mean predicted FVC was 78.9 ± 0.5%. Forty-two patients met the criteria for severe disease, and 18 patients died. Mortality was significantly associated with increased CO exposure (for each 0.1 mg/m2 increase: odds ratio 2.45, 95% confidence interval 1.39–4.56). We observed no association between any of the other investigated contaminants and IPF mortality or severity. Conclusions Air pollution, specifically that caused by carbon monoxide, can increase mortality in patients with IPF.