Collision-induced dissociation (CID) remains the predominant mass spectrometry based method for identifying phosphorylation sites in complex mixtures. Unfortunately, the gas-phase reactivity of phosphoester bonds results in MS/MS spectra dominated by phosphoric acid (H3PO4) neutral loss events, suppressing informative peptide backbone cleavages. To understand the major drivers of H3PO4 neutral loss, we performed robust non-parametric statistical analysis of local and distal sequence effects on the magnitude and variability of neutral loss, using a collection of over 35,000 unique phosphopeptide MS/MS spectra. In contrast to peptide amide dissociation pathways, which are strongly influenced by adjacent amino acid side chains, we find that neutral loss of H3PO4 is affected by both proximal and distal sites, most notably basic residues and the peptide N-terminal primary amine. Previous studies have suggested that protonated basic residues catalyze neutral loss through direct interactions with the phosphate. In contrast, we find that nearby basic groups decrease neutral loss regardless of mobility class, an effect only seen by stratifying spectra by charge-mobility. The most inhibitory bases are those immediately N-terminal to the phosphate, presumably due to steric hindrances in catalyzing neutral loss. Further evidence of steric effects is shown by the presence of proline which can dramatically reduce the presence of neutral loss when between the phosphate and a possible charge donor. In mobile proton spectra the N-terminus is the strongest predictor of high neutral loss, with proximity to the N-terminus essential for peptides to exhibit the highest levels of neutral loss.