We examined the association between face masks and risk of infection with SARS-CoV-2 using cross-sectional data from 3,209 participants in a randomized trial exploring the effectiveness of glasses in reducing the risk of SARS-CoV-2 infection. Face mask use was based on participants’ response to the end-of-follow-up survey. We found that the incidence of self-reported COVID-19 was 33% (aRR 1.33; 95% CI 1.03–1.72) higher in those wearing face masks often or sometimes, and 40% (aRR 1.40; 95% CI 1.08–1.82) higher in those wearing face masks almost always or always, compared to participants who reported wearing face masks never or almost never. We believe the observed increase in the incidence of infection associated with wearing a face mask is likely due to unobservable and hence nonadjustable differences between those wearing and not wearing a mask. Observational studies reporting on the relationship between face mask use and risk of respiratory infections should be interpreted cautiously, and more randomized trials are needed.