This paper characterizes the annealed, topological complexity (both of total critical points and of local minima) of the elastic manifold. This classical model of a disordered elastic system captures point configurations with self‐interactions in a random medium. We establish the simple versus glassy phase diagram in the model parameters, with these phases separated by a physical boundary known as the Larkin mass, confirming formulas of Fyodorov and Le Doussal. One essential, dynamical, step of the proof also applies to a general signal‐to‐noise model of soft spins in an anisotropic well, for which we prove a negative‐second‐moment threshold distinguishing positive from zero complexity. A universal near‐critical behavior appears within this phase portrait, namely quadratic near‐critical vanishing of the complexity of total critical points, and cubic near‐critical vanishing of the complexity of local minima. These two models serve as a paradigm of complexity calculations for Gaussian landscapes exhibiting few distributional symmetries, that is, beyond the invariant setting. The two main inputs for the proof are determinant asymptotics for non‐invariant random matrices from our companion paper (Ben Arous, Bourgade, McKenna 2022), and the atypical convexity and integrability of the limiting variational problems.