Quantum emitters coupled to plasmonic nanostructures can act as exceptionally bright sources of single photons, operating at room temperature. Plasmonic mode volumes supported by these nanostructures can be several orders of magnitude smaller than the cubic wavelength, which leads to dramatically enhanced light-matter interactions and drastically increased photon production rates. However, when increasing the light localization further, these deeply subwavelength modes may in turn hinder the fast outcoupling of photons into free space. Plasmonic hybrid nanostructures combining a highly confined cavity mode and a larger antenna mode circumvent this issue. We establish the fundamental limits for quantum emission enhancement in such systems and find that the best performance is achieved when the cavity and antenna modes differ significantly in size. We experimentally support this idea by photomodifying a nanopatch antenna deterministically assembled around a nanodiamond known to contain a single nitrogen-vacancy (NV) center. As a result, the cavity mode shrinks, further shortening the NV fluorescence lifetime and increasing the single-photon brightness. Our analytical and numerical simulation results provide intuitive insight into the operation of these emitter-cavity-antenna systems and show that this approach could lead to single-photon sources with emission rates up to hundreds of THz and efficiencies close to unity.