With a highly coherent, optically addressable electron spin, the nitrogen vacancy (NV) centre in diamond is a promising candidate for a node in a quantum network. However, the NV centre is a poor source of coherent single photons owing to a long radiative lifetime, a small branching ratio into the zero-phonon line (ZPL) and a poor extraction efficiency out of the high-index host material. In principle, these three shortcomings can be addressed by resonant coupling to a single mode of an optical cavity. Utilising the weak-coupling regime of cavity electrodynamics, resonant coupling between the ZPL and a single cavity-mode enhances the transition rate and branching ratio into the ZPL. Furthermore, the cavity channels the light into a well-defined mode thereby facilitating detection with external optics. Here, we present an open Fabry-Perot microcavity geometry containing a single-crystal diamond membrane, which operates in a regime where the vacuum electric field is strongly confined to the diamond membrane. There is a field anti-node at the diamond-air interface. Despite the presence of surface losses, quality factors exceeding 120 000 and a finesse F = 11 500 were observed. We investigate the interplay between different loss mechanisms, and the impact these loss channels have on the performance of the cavity. This analysis suggests that the "waviness" (roughness with a spatial frequency comparable to that of the microcavity mode) is the mechanism preventing the quality factors from reaching even higher values. Finally, we apply the extracted cavity parameters to the NV centre and calculate a predicted Purcell factor exceeding 150.