Protein–protein interactions that involve recognition of short peptides are critical in cellular processes. Protein–peptide interaction surface areas are relatively small and shallow, and there are often overlapping specificities in families of peptide‐binding domains. Therefore, dissecting selectivity determinants can be challenging. PDZ domains are a family of peptide‐binding domains located in several intracellular signaling and trafficking pathways. These domains are also directly targeted by pathogens, and a hallmark of many oncogenic viral proteins is a PDZ‐binding motif. However, amidst sequences that target PDZ domains, there is a wide spectrum in relative promiscuity. For example, the viral HPV16 E6 oncoprotein recognizes over double the number of PDZ domain‐containing proteins as the cystic fibrosis transmembrane conductance regulator (CFTR) in the cell, despite similar PDZ targeting‐sequences and identical motif residues. Here, we determine binding affinities for PDZ domains known to bind either HPV16 E6 alone or both CFTR and HPV16 E6, using peptides matching WT and hybrid sequences. We also use energy minimization to model PDZ–peptide complexes and use sequence analyses to investigate this difference. We find that while the majority of single mutations had marginal effects on overall affinity, the additive effect on the free energy of binding accurately describes the selectivity observed. Taken together, our results describe how complex and differing PDZ interactomes can be programmed in the cell.