We proposed a double-referential holography (DRH) that allows phase-detection without external additional beams. In the DRH, phantom beams, prepared in the same optical path as signal beams and preliminary multiplexed in a recording medium along with the signal, are used to produce interference fringes on an imager for converting a phase into an intensity distribution. The DRH enables stable and high-accuracy phase detection independent of the fluctuations and vibrations of the optical system owing to medium shift and temperature variation. Besides, the collinear arrangement of the signal and phantom beams leads to the compactness of the optical data storage system. We conducted an experiment using binary phase modulation signals for verifying the DRH operation. In addition, 38-level spatial quadrature amplitude modulation signals were successfully reproduced with the DRH by numerical simulation. Furthermore, we verified that the distributed phase-shifting method moderates the dynamic range consumption for the exposure of phantom beams.