In two dimensions, magnetic higher-order topological insulators (HOTIs) are characterized by excess boundary charge and a compensating bulk "filling anomaly." At the same time, without additional noncrystalline symmetries, the boundaries of two-dimensional HOTIs are gapped and featureless at low energies, while the bulk of the system is predicted to have a topological response to the insertion of lattice (particularly disclination) defects. Until recently, a precise connection between these effects has remained elusive. In this work, we provide a unifying field-theoretic description for the bulk and boundary response of magnetic HOTIs. By focusing on the low-energy description of the gapped boundary of a HOTI, we show that the boundary charge and filling anomaly arise from the gravitational "Gromov-Jensen-Abanov" (GJA) response action first introduced in [Phys. Rev. Lett. 116, 126802 (2016)] in the context of the quantum Hall effect. As in quantum Hall systems the GJA action cancels apparent anomalies associated with bulk response to disclinations, allowing us to derive a concrete connection between the bulk and boundary theories of HOTIs. We show how our results elucidate the connection between higher order topology and geometric response both in band insulators, and point towards a new route to understanding interacting higher order topological phases.