Multi-zinc finger proteins are the largest class of human transcription factors with integral roles in genome regulation and function. Often using a subset of their zinc finger domains, these proteins bind to diverse DNA sequences that make up the majority of the human regulatory lexicon. However, the molecular code that underlies the interaction of the zinc finger with DNA is incompletely understood, and for most multi-zinc finger proteins the zinc finger subset that is responsible for in vivo DNA binding is unknown. Here, we present a computational model, derived from a compendium of in vivo and in vitro binding preferences, that captures context-specific binding of zinc fingers to DNA. This model can predict the binding specificity of each zinc finger in its natural context more accurately than existing models, and together with molecular dynamics analyses reveals new structural aspects of DNA recognition by zinc finger proteins, including novel amino acid residues that contribute to sequence specificity. Furthermore, by combining this computational model with in vivo binding data, we identify the sequence preference of each protein and the zinc finger subset that is responsible for in vivo DNA binding, providing DNA binding maps and the associated domains for ~30% of all human multi-zinc finger proteins. We show that compared to non-DNA-binding zinc finger domains, DNA-binding zinc fingers are under stronger selective pressure across species and are depleted of genetic variants in the human population. Finally, we propose that a combination of context as well as intrinsic zinc finger features determines the ability to bind to DNA in vivo.