Background Aspergillus fumigatus is the most common filamentous fungus isolated from the airways of people with cystic fibrosis (CF). The aim of this study was to investigate how chronic A. fumigatus colonization affects lung function in people with CF, to identify risk factors for colonization, and to evaluate antifungal treatment of asymptomatic Aspergillus colonization. Methods Data from 2014–2018 was collected from the Swedish CF registry and medical records. Baseline data before the start of A. fumigatus colonization was compared with the two succeeding years to evaluate how colonization and treatment affected lung function and other clinical aspects. Results A total of 437 patients were included, of which 64 (14.6%) became colonized with A. fumigatus during the study period. Inhaled antibiotics was associated with A. fumigatus colonization (adjusted OR 3.1, 95% CI 1.6–5.9, p < 0.05). Fungal colonization was not associated with a more rapid lung function decline or increased use of IV-antibiotics compared to the non-colonized group, but patients with A. fumigatus had more hospital days, a higher increase of total IgE, and higher eosinophil counts. In the Aspergillus group, 42 patients were considered to be asymptomatic. Of these, 19 patients received antifungal treatment. Over the follow up period, the treated group had a more pronounced decrease in percent predicted Forced Expiratory Volume in one second (ppFEV1) compared to untreated patients (− 8.7 vs − 1.4 percentage points, p < 0.05). Conclusion Inhaled antibiotics was associated with A. fumigatus colonization, but no association was found between persistent A. fumigatus and subsequent lung function decline. No obvious benefits of treating asymptomatic A. fumigatus colonization were demonstrated.