We investigate the integrability anomalies arising in the self-dual sectors of gravity and Yang-Mills theory, focusing on their connection to both the chiral anomaly and the trace anomaly. The anomalies in the self-dual sectors generate the one-loop all-plus amplitudes of gravitons and gluons, and have recently been studied via twistor constructions. On the one hand, we show how they can be interpreted as an anomaly of the chiral U(1) electric-magnetic-type duality in the self-dual sectors. We also note the similarity, for the usual fermionic chiral anomaly, between the 4D setting of self-dual Yang-Mills and the 2D setting of the Schwinger model. On the other hand, the anomalies in the self-dual theories also resemble the trace anomaly, sharing the same type of non-local effective action. We highlight the role of a Weyl-covariant fourth-order differential operator familiar from the trace anomaly literature, which (i) explains the conformal properties of the one-loop amplitudes, and (ii) indicates how this story may be extended to non-trivial spacetime backgrounds, e.g. with a cosmological constant. Moving beyond the self-dual sectors, and focusing on the gravity case, we comment on an intriguing connection to the two-loop ultraviolet divergence of pure gravity, whereby cancelling the anomaly at one-loop eliminates the two-loop divergence for the simplest helicity amplitudes.