The identification of mutations that enhance antibody affinity while maintaining high antibody specificity and stability is a time-consuming and laborious process. Here, we report an efficient methodology for systematically and rapidly enhancing the affinity of antibody variable domains while maximizing specificity and stability using novel synthetic antibody libraries. Our approach first uses computational and experimental alanine scanning mutagenesis to identify sites in the complementarity-determining regions (CDRs) that are permissive to mutagenesis while maintaining antigen binding. Next, we mutagenize the most permissive CDR positions using degenerate codons to encode wild-type residues and a small number of the most frequently occurring residues at each CDR position based on natural antibody diversity. This mutagenesis approach results in antibody libraries with variants that have a wide range of numbers of CDR mutations, including antibody domains with single mutations and others with tens of mutations. Finally, we sort the modest size libraries (~10 million variants) displayed on the surface of yeast to identify CDR mutations with the greatest increases in affinity. Importantly, we find that single-domain (VHH) antibodies specific for the α-synuclein protein (whose aggregation is associated with Parkinson’s disease) with the greatest gains in affinity (>5-fold) have several (four to six) CDR mutations. This finding highlights the importance of sampling combinations of CDR mutations during the first step of affinity maturation to maximize the efficiency of the process. Interestingly, we find that some natural diversity mutations simultaneously enhance all three key antibody properties (affinity, specificity, and stability) while other mutations enhance some of these properties (e.g., increased specificity) and display trade-offs in others (e.g., reduced affinity and/or stability). Computational modeling reveals that improvements in affinity are generally not due to direct interactions involving CDR mutations but rather due to indirect effects that enhance existing interactions and/or promote new interactions between the antigen and wild-type CDR residues. We expect that natural diversity mutagenesis will be useful for efficient affinity maturation of a wide range of antibody fragments and full-length antibodies.