Extreme mass ratio inspirals, in which a stellar-mass object orbits a supermassive black hole, are prime sources for space-based gravitational wave detectors because they will facilitate tests of strong gravity and probe the spacetime around rotating compact objects. In the last few years of such inspirals, the total phase is in the millions of radians and details of the waveforms are sensitive to small perturbations. We show that one potentially detectable perturbation is the presence of a second supermassive black hole within a few tenths of a parsec. The acceleration produced by the perturber on the extreme mass-ratio system produces a steady drift that causes the waveform to deviate systematically from that of an isolated system. If the perturber is a few tenths of a parsec from the extreme-mass ratio system (plausible in as many as a few percent of cases) higher derivatives of motion might also be detectable. In that case, the mass and distance of the perturber can be derived independently, which would allow a new probe of merger dynamics.