Low temperature polarized single-crystal absorption and luminescence spectra of Cr(V)-doped YVO4 and the powder luminescence spectrum of Cr(V)-doped YPO4 are reported and discussed. A rich fine structure and strong polarization effects are observed in the near-infrared. Due to a strong interaction of the Cr(V) ion with two Y3+ ions in the second coordination sphere of the Cr(V) ion, the electronic ground state is different from the one expected on the basis of an angular overlap calculation in which only the four oxygen ligands are taken into account. This effect of the Y3+ ions on the ground state of Cr(V) is confirmed by a density functional calculation and by literature EPR data. CrO4(3-) bending modes are responsible for the fine structure in the d-d transition and the resulting distortion in the emitting excited state.