COVID-19 has had an unprecedented global impact on human health. Understanding the antibody memory responses to infection is one tool needed to effectively control the pandemic. Among 173 outpatients who had virologically confirmed SARS-CoV-2 infection, we evaluated serum antibody concentrations, microneutralization activity, and enumerated SARS-CoV-2 specific B cells in convalescent blood specimens. Serum antibody concentrations were variable, allowing for stratification of the cohort into high and low responders. Serum antibody concentration was positively associated with microneutralization activity and participant age, with participants under the age of 30 showing the lowest antibody level. Neither participant sex, the timing of blood sampling following the onset of illness, nor the number of SARS-CoV-2 spike protein specific B cells correlated with serum antibody concentration. These data suggest that young adult outpatients did not generate as robust antibody memory, compared with older adults. Further, serum antibody concentration or neutralizing activity trended but did not significantly correlate with the number of SARS-CoV-2 memory B cells. These findings have direct implications for public health policy and current vaccine efforts. Knowledge gained regarding antibody memory following infection will inform the need for vaccination in those previously infected and allow for a better approximation of population-wide protective immunity.