We show that, under weak assumptions, the automorphism group of a CAT(0) cube complex X coincides with the automorphism group of Hagen's contact graph C(X). The result holds, in particular, for universal covers of Salvetti complexes, where it provides an analogue of Ivanov's theorem on curve graphs of non-sporadic surfaces. This highlights a contrast between contact graphs and Kim-Koberda extension graphs, which have much larger automorphism group.We also study contact graphs associated to Davis complexes of rightangled Coxeter groups. We show that these contact graphs are less well-behaved and describe exactly when they have more automorphisms than the universal cover of the Davis complex.