We consider the possibility for phases of matter in which a continuous symmetry is spontaneously broken in a topologically non-trivial way, which, roughly, means that the action for the Goldstone modes contains a quantized topological term, and could manifest in, for example, non-trivial quantum numbers of topological defects of the order parameter. We show that, in fact, such a scenario can occur only when the system is in a non-trivial symmetry-protected topological (SPT) or symmetryenriched topological (SET) phase with respect to the residual symmetry; or alternatively, if the original symmetry before spontaneous symmetry breaking acts on the system in an "anomalous" way. Our arguments are based on a general correspondence between topological defects of the order parameter and topological defects of a background gauge field for the residual symmetry.