The contribution of heritable factors to antibody function and diversity is not fully understood, but has profound implications for delineating variation in the antibody response observed at the population-level. We performed matched long-read-based characterization of the immunoglobulin heavy chain (IGH) locus and expressed antibody repertoire profiling at population-scale to examine, for the first time, the impact of IGH genomic variation on the antibody repertoire. We characterized extensive IGH polymorphism, including novel structural variants (SVs), small insertion/deletions (indels), single nucleotide variants (SNVs), and IG genes and alleles. Countering models that antibody repertoire diversity is driven largely by stochastic processes, we demonstrate that IGH genetic factors make significant contributions to gene usage in both the naive and antigen-experienced repertoire. Specifically, the usage of 73% of IGH genes was associated with common polymorphisms, including those capable of explaining >70% of variance in gene usage. These variants were enriched in transcription factor binding sites and other functional elements associated with V(D)J recombination, and overlapped polymorphisms from genome-wide association studies. Furthermore, we found evidence for the coordinated regulation of IGH genes across the repertoire, demonstrating complex interactions between IGH variants and gene usage. These results refine our understanding of variation observed in the antibody repertoire, and will advance the study of antibody function in disease.