Antibodies raised against highly prevalent human seasonal coronaviruses (sCoVs), which are responsible for the common cold, are known to cross-react with SARS-CoV-2 antigens. This cross-reactivity prompts questions about their protective role against SARS-CoV-2 infections and COVID-19 disease severity. However, the relationship between sCoV exposure and SARS-CoV-2 correlates of protection have not been clearly identified. Here we performed a cross-sectional analysis of cross-reactivity and cross-neutralization to three SARS-CoV-2 antigens using pre-pandemic serum from four different groups: pediatrics and adolescents (<21 yrs of age), persons 21 to 70 yrs of age, persons older than 70 yrs of age, and persons living with HCV or HIV. We find that antibody cross-reactivity to SARS-CoV-2 antigens varied between 1.6% and 15.3% depending on the cohort and the isotype-antigen pair analyzed. We also demonstrate a broad range of neutralizing activity (0-45%) in pre-pandemic serum that interferes with SARS-CoV-2 spike attachment to ACE2. While the abundance of sCoV antibodies did not directly correlate with neutralization efficiency, by using machine learning methodologies, we show that neutralizing activity is rather dependent on the latent variables related to the pattern ratios of sCoVs antibodies presented by each person. These were independent of age or sex, and could be accurately predicted by comparing the relative ratios of IgGs in sera directed to NL63, 229E, HKU-1, and OC43 spike proteins. More specifically, we identified antibodies to NL63 and OC43 as being the two most important predictors of latent variables responsible for protection, and 229E as being the least weighted. Our data support that exposure to sCoVs triggers various cellular and immune responses that influence the efficiency of SARS-CoV-2 spike binding to ACE2, and may impact COVID-19 disease severity through various other latent variables.