Thin films of ferromagnetic semiconductors of the (Ga, Mn)(As, P) family are investigated by means of magneto-optical Kerr spectroscopy in order to determine the spectral dependence of the complex magneto-optical Voigt constant Q, an intrinsic parameter responsible for Kerr rotation and ellipticity. We obtain the spectral dependence of these two conjugate quantities: the Kerr rotation and ellipticity angles. Contrary to strongly absorbing metallic ferromagnetic layers, it is expected that they vary non-linearly with the layer thickness at a fixed wavelength, as a consequence of the weak absorption and the non-negligible optical phase shift in weakly absorbing semiconductor ferromagnets. We show that, from the Kerr rotation and ellipticity angles and the complex refractive index-obtained by ellipsometric measurements-we are able to determine the spectral dependence of the complex intrinsic Voigt constant, thus avoiding the need for samples with different thicknesses. Incidentally, the use of phosphorus substituted (Ga, Mn)As provides an out-of-plane magnetic easy axis, the appropriate configuration for polar magneto-optical Kerr effect, without the need for a strong external magnetic field.