Optical cavities can support many transverse and longitudinal modes. A paraxial scalar theory predicts that the resonance frequencies of these modes cluster in different orders. A non-paraxial vector theory predicts that the frequency degeneracy within these clusters is lifted, such that each order acquires a spectral fine structure, comparable to the fine structure observed in atomic spectra. In this paper, we calculate this fine structure for microcavities and show how it originates from various non-paraxial effects and is co-determined by mirror aberrations. The presented theory, which applies perturbation theory to Maxwell's equations with boundary conditions, proves to be very powerful. It generalizes the effective 1-dimensional description of Fabry-Perot cavities to a 3dimensional multi-transverse-mode description. It thereby provides new physical insights in several mode-shaping effects and a detailed prediction of the fine structure in Fabry-Perot spectra.