Roughness in graphene is known to contribute to scattering effects which lower carrier mobility. Encapsulating graphene in hexagonal boron nitride (hBN) leads to a significant reduction in roughness and has become the de facto standard method for producing high-quality graphene devices. We have fabricated graphene samples encapsulated by hBN that are suspended over apertures in a substrate and used noncontact electron diffraction measurements in a transmission electron microscope to measure the roughness of encapsulated graphene inside such structures. We furthermore compare the roughness of these samples to suspended bare graphene and suspended graphene on hBN. The suspended heterostructures display a root mean square (rms) roughness down to 12 pm, considerably less than that previously reported for both suspended graphene and graphene on any substrate and identical within experimental error to the rms vibrational amplitudes of carbon atoms in bulk graphite. Our first-principles calculations of the phonon bands in graphene/hBN heterostructures show that the flexural acoustic phonon mode is localized predominantly in the hBN layer. Consequently, the flexural displacement of the atoms in the graphene layer is strongly suppressed when it is supported by hBN, and this effect increases when graphene is fully encapsulated.