Arrestins bind active phosphorylated forms of G protein-coupled receptors, terminating G protein activation, orchestrating receptor trafficking, and redirecting signaling to alternative pathways. Visual arrestin-1 preferentially binds rhodopsin, whereas the two non-visual arrestins interact with hundreds of G protein-coupled receptor subtypes. Here we show that an extensive surface on the concave side of both arrestin-2 domains is involved in receptor binding. We also identified a small number of residues on the receptor binding surface of the N-and C-domains that largely determine the receptor specificity of arrestins. We show that alanine substitution of these residues blocks the binding of arrestin-1 to rhodopsin in vitro and of arrestin-2 and -3 to 2-adrenergic, M2 muscarinic cholinergic, and D2 dopamine receptors in intact cells, suggesting that these elements critically contribute to the energy of the interaction. Thus, in contrast to arrestin-1, where direct phosphate binding is crucial, the interaction of non-visual arrestins with their cognate receptors depends to a lesser extent on phosphate binding and more on the binding to non-phosphorylated receptor elements.