A large number of small membrane proteins have been discovered in bacteria, but their mechanism of action has remained mostly elusive. Here, we investigate the mechanism of a physiologically important small protein, MgrB, which represses the activity of the sensor kinase PhoQ and is widely distributed among enterobacteria. The PhoQ/PhoP two-component system is a master regulator of the bacterial virulence program and interacts with MgrB to modulate bacterial virulence, fitness, and drug resistance. A combination of crosslinking approaches with functional assays and protein dynamic simulations revealed structural rearrangements due to interactions between MgrB and PhoQ near the membrane/periplasm interface and along the transmembrane helices. These interactions induce the movement of the PhoQ catalytic domain and the repression of its activity. In the absence of MgrB, PhoQ appears to be insensitive to antimicrobial peptides, including the commonly used C18G. MgrB mediates the sensing of C18G by dissociating from and thus derepressing PhoQ. Our findings reveal the inhibitory mechanism of the small protein MgrB and uncover its importance in antimicrobial peptide sensing.