After the first pandemic wave, a nationwide survey assessed the seroprevalence of SARS-CoV-2 antibodies in Spain and found notable differences among provinces whose causes remained unclear. This ecological study aimed to analyze the association between environmental and demographic factors and SARS-CoV-2 infection by province. The seroprevalence of SARS-CoV-2 antibodies by province was obtained from a nationwide representative survey performed in June 2020, after the first pandemic wave in Spain. Linear regression was used in the analysis. The seroprevalence of SARS-CoV-2 antibodies of the 50 provinces ranged from 0.2% to 13.6%. The altitude, which ranged from 5 to 1131 m, explained nearly half of differences in seroprevalence (R2 = 0.47, p < 0.001). The seroprevalence in people residing in provinces above the median altitude (215 m) was three-fold higher (6.5% vs. 2.1%, p < 0.001). In the multivariate linear regression, the addition of population density significantly improved the predictive value of the altitude (R2 = 0.55, p < 0.001). Every 100 m of altitude increase and 100 inhabitants/km2 of increase in population density, the seroprevalence rose 0.84 and 0.63 percentage points, respectively. Environmental conditions related to higher altitude in winter–spring, such as lower temperatures and absolute humidity, may be relevant to SARS-CoV-2 transmission. Places with such adverse conditions may require additional efforts for pandemic control.