Context. The detection and characterization of Earth-like exoplanets is one of the major science drivers for the next generation of telescopes. Direct imaging of the planets will play a major role in observations. Current direct imaging instruments are limited by evolving non-common path aberrations (NCPAs). The NCPAs have to be compensated for by using the science focal-plane image. A promising sensor is the self-coherent camera (SCC). An SCC adds a pinhole to the Lyot stop in the coronagraph to introduce a probe electric field. The pinhole has to be separated by at least 1.5 times the pupil size to separate the NCPA speckles from the probe electric field. However, such a distance lets through very little light, which makes it difficult to use an SCC at high speed or on faint targets. Aims. A spectrally modulated self-coherent camera (SM-SCC) is proposed as a solution to the throughput problem. The SM-SCC uses a pinhole with a spectral filter and a dichroic beam splitter, which creates images with and without the probe electric field. This allows the pinhole to be placed closer to the pupil edge and increases the throughput. Combining the SM-SCC with an integral field unit (IFU) can be used to apply more complex modulation patterns to the pinhole and the Lyot stop. A modulation scheme with at least three spectral channels can be used to change the pinhole to an arbitrary aperture with higher throughput. This adds an additional degree of freedom in the design of the SM-SCC. Methods. The performance of the SM-SCC is investigated analytically and through numerical simulations.Results. Numerical simulations show that the SM-SCC increases the pinhole throughput by a factor of 32, which increases the wavefront sensor sensitivity by a factor of 5.7. The reconstruction quality of the sensor is tested by varying the central wavelength of the spectral channels. A smaller separation between the wavelength channels leads to better results. The SM-SCC reaches a contrast of 1•10 −9 for bright targets in closed-loop control with the presence of photon noise, phase errors, and amplitude errors. The contrast floor on fainter targets is photon-noise-limited and reaches 1 • 10 −7 . The SM-SCC with an IFU can handle randomly generated reference field apertures. For bright targets, the SM-SCC-IFU reaches a contrast of 3 • 10 −9 in closed-loop control with photon noise, amplitude errors, and phase errors. Conclusions. The SM-SCC is a promising focal-plane wavefront sensor for systems that use multiband observations, either through integral field spectroscopy or dual-band imaging.