Context. The fringe sensor unit (FSU) is the central element of the phase referenced imaging and micro-arcsecond astrometry (PRIMA) dual-feed facility and provides fringe sensing for all observation modes, comprising off-axis fringe tracking, phase referenced imaging, and high-accuracy narrow-angle astrometry. It is installed at the Very Large Telescope Interferometer (VLTI) and successfully served the fringe-tracking loop during the initial commissioning phase. Aims. To maximise sensitivity, speed, and robustness, the FSU is designed to operate in the infrared K-band and to include spatial filtering after beam combination and a very-low-resolution spectrometer without photometric channels. It consists of two identical fringe sensors for dual-star operation in PRIMA astrometric mode. Methods. Unique among interferometric beam combiners, the FSU uses spatial phase modulation in bulk optics to retrieve real-time estimates of fringe phase after spatial filtering. The beam combination design accommodates a laser metrology for pathlength monitoring. An R = 20 spectrometer across the K-band makes the retrieval of the group delay signal possible. The calibration procedure uses the artificial light source of the VLTI laboratory and is based on Fourier transform spectroscopy to remove instrumental effects. Results. The FSU was integrated and aligned at the VLTI in July and August 2008. It yields phase and group delay measurements at sampling rates up to 2 kHz, which are used to drive the fringe-tracking control loop. During the first commissioning runs, the FSU was used to track the fringes of stars with K-band magnitudes as faint as m K = 9.0, using two VLTI auxiliary telescopes (AT) and baselines of up to 96 m. Fringe tracking using two Very Large Telescope (VLT) unit telescopes was demonstrated. Conclusions. The concept of spatial phase-modulation for fringe sensing and tracking in stellar interferometry is demonstrated for the first time with the FSU. During initial commissioning and combining stellar light with two ATs, the FSU showed its ability to improve the VLTI sensitivity in K-band by more than one magnitude towards fainter objects, which is fundamental for achieving the scientific objectives of PRIMA.