Phosphorylation of the hydroxyl group of the amino acids serine and threonine is among the most prevalent post-translational modifications in mammalian cells. Phospho-serine (pSer) and -threonine (pThr) represent a central cornerstone in the cell's toolbox for adaptation to signal input. The true power for the fast modulation of the regulatory pSer/pThr sites arises from the timely attachment, binding and removal of the phosphate. The phosphorylation of serine and threonine by kinases and the binding of pSer/pThr by phosphorylation-dependent scaffold proteins is largely determined by the sequence motif surrounding the phosphorylation site (p-site). The removal of the phosphate is regulated by pSer/pThr-specific phosphatases with the two most prominent ones being PP1 and PP2A. For this family, recent advances brought forward a more complex mechanism for p-site selection. The interaction of regulatory proteins with the substrate protein constitutes a first layer for substrate recognition, but also interactions of the catalytic subunit with the amino acids in close proximity to pSer/pThr contribute to p-site selection. Here, we review the current pieces of evidence for this multi-layered, complex mechanism and hypothesize that, depending on the degree of higher structure surrounding the substrate site, recognition is more strongly influenced by regulatory subunits away from the active site for structured substrate regions, whereas the motif context is of strong relevance with p-sites in disordered regions. The latter makes these amino acid sequences crossroads for signaling and motif strength between kinases, pSer/pThr-binding proteins and phosphatases.