Non-Fermi liquid phenomena arise naturally near critical points of Landau ordering transitions in metallic systems, where strong fluctuations of a bosonic order parameter destroy coherent quasiparticles. Despite progress in developing controlled perturbative techniques, much of the low energy physics of such metallic quantum critical points remains poorly understood. We demonstrate that exact, non-perburbative results can be obtained for both optical transport and static susceptibilities in "Hertz-Millis" theories of Fermi surfaces coupled to critical bosons. Such models possess a large emergent symmetry and anomaly structure, which we leverage to fix these quantities. In particular, we show that in the infrared limit, the boson self energy at zero wave vector, q = 0, is a constant independent of frequency, and the real part of the optical conductivity, σ(ω), is purely a delta function Drude peak with no other corrections. Therefore, further frequency dependence in the boson self energy or optical conductivity can only come from irrelevant operators in a clean system. Exact relations between Fermi liquid parameters as the critical point is approached from the disordered phase are also obtained. The absence of a universal, power law frequency dependence in the boson self energy contrasts with previous perturbative calculations, and we explain the origin of this discrepancy.