Phage-displayed immunoprecipitation sequencing (PhIP-Seq) has successfully enabled high-throughput profiling of human antibody profiles. However, a comprehensive overview of environmental and genetic determinants shaping human adaptive immunity is currently lacking. In this study, we aimed to investigate the effects of genetic, environmental and intrinsic factors on the variation in human antibody repertoires. We characterized serological antibody repertoires against 344,000 peptides using PhIP-Seq libraries from a wide range of microbial and environmental antigens in 1,443 participants from a population cohort. We demonstrate individual-specificity, temporal consistency and co-housing similarities in antibody repertoire. Genetic analyses showed involvement of the HLA, IGHV and FUT2 regions. Furthermore, we uncovered associations between 48 phenotypic factors and 544 antibody-bound peptides, including age, cell counts, sex, smoking behavior and allergies, among others. Overall, our results indicate that human antibody epitope repertoires are shaped by both host genetics and environmental exposures and highlight unique signatures of distinct phenotypes and genotypes.