Holographic displays have great potential to realize mixed reality by modulating the wavefront of light in a fundamental manner. As a computational display, holographic displays offer a large degree of freedom, such as focus cue generation and vision correction. However, the limited bandwidth of spatial light modulator imposes an inherent trade-off relationship between the field of view and eye-box size. Thus, we demonstrate the first practical eye-box expansion method for a holographic near-eye display. Instead of providing an intrinsic large exit-pupil, we shift the optical system's exit-pupil to cover the expanded eye-box area with pupil-tracking. For compact implementation, a pupil-shifting holographic optical element (PSHOE) is proposed that can reduce the form factor for exit-pupil shifting. A thorough analysis of the design parameters and display performance are provided. In particular, we provide a comprehensive analysis of the incorporation of the holographic optical element into a holographic display system. The influence of holographic optical elements on the intrinsic exit-pupil and pupil switching is revealed by numerical simulation and Wigner distribution function analysis.