Background: Direct and continuous exposure to particulate matter (PM), especially in occupational settings is known to impact negatively on respiratory health and lung function. Objective: To determine the association between concentrations of PM (2.5, 2.5–10 and 10 µm) in breathing zone and lung function of informal e-waste workers at Agbogbloshie. Methods: To evaluate lung function responses to PM (2.5, 2.5–10 and 10 µm), we conducted a longitudinal cohort study with three repeated measures among 207 participants comprising 142 healthy e-waste workers from Agbogbloshie scrapyard and 65 control participants from Madina-Zongo in Accra, Ghana from 2017–2018. Lung function parameters (FVC, FEV1, FEV1/FVC, PEF, and FEF 25-75) and PM (2.5, 2.5–10 and 10 µm) concentrations were measured, corresponding to prevailing seasonal variations. Socio-demographic data, respiratory exposures and lifestyle habits were determined using questionnaires. Random effects models were then used to examine the effects of PM (2.5, 2.5–10 and 10 µm) on lung function. Results: The median concentrations of PM (2.5, 2.5–10 and 10 µm) were all consistently above the WHO ambient air standards across the study waves. Small effect estimates per IQR of PM (2.5, 2.5–10 and 10 µm) on lung function parameters were observed even after adjustment for potential confounders. However, a 10 µg increase in PM (2.5, 2.5–10 and 10 µm) was associated with decreases in PEF and FEF 25–75 by 13.3% % [β = −3.133; 95% CI: −0.243, −0.022) and 26.6% [β = −0.266; 95% CI: −0.437, 0.094]. E-waste burning and a history of asthma significantly predicted a decrease in PEF by 14.2% [β = −0.142; 95% CI: −0.278, −0.008) and FEV1 by 35.8% [β = −0.358; 95% CI: −0.590, 0.125] among e-waste burners. Conclusions: Direct exposure of e-waste workers to PM predisposes to decline in lung function and risk for small airway diseases such as asthma and COPD.