In this paper, a compact balanced diplexer using a novel hairpin split ring resonator (H-SRR) is presented and demonstrated. First, the working principle of the proposed H-SRR is described, which shows a negative permittivity or a negative permeability in the stopband. It can be used to construct compact passive components and improve the stopband rejection performance. Then, the differential-mode (DM) excitation and common-mode (CM) excitation are investigated, respectively. Under DM operation, the H-SRR can provide one DM resonant mode. Under CM operation, the H-SRR can excite two CM resonant modes far from the DM resonant mode, thus producing a desired CM rejection performance. Moreover, by introducing a mixed electromagnetic coupling, transmission zeros are produced, significantly improving the DM isolation between the two channels. Finally, a balanced diplexer is designed and fabricated. The lower and higher channels of the diplexer are centered at 3.36 and 4.00 GHz. The DM channel isolation is better than 40/41 dB in the two passbands when the frequency ratio is less than 1.2, which is in satisfactory agreement with the simulated results.