High spectral mode density of conventional optical cavities is detrimental to the generation of broad optical frequency combs and to other linear and nonlinear applications. In this work we optimize the morphology of high-Q whispering gallery (WG) and Fabry-Perot (FP) cavities and find a set of parameters that allows treating them, essentially, as single-mode structures, thus removing limitations associated with a high density of cavity mode spectra. We show that both single-mode WGs and single-mode FP cavities have similar physical properties, in spite of their different loss mechanisms. The morphology optimization does not lead to a reduction of quality factors of modes belonging to the basic family. We study the parameter space numerically and find the region where the highest possible Q factor of the cavity modes can be realized while just having a single bound state in the cavity. The value of the Q factor is comparable with that achieved in conventional cavities. The proposed cavity structures will be beneficial for generation of octave spanning coherent frequency combs and will prevent undesirable effects of parametric instability in laser gravitational wave detectors.