We combine hydrodynamic and field theoretic methods to develop a general theory of phonons as Goldstone bosons in crystals, glasses, and liquids based on nonaffine displacements and the consequent Goldstone phase relaxation. We relate the conservation, or lack thereof, of specific higher-form currents with properties of the underlying deformation field-nonaffinity-which dictates how molecules move under an applied stress or deformation. In particular, the single-valuedness of the deformation field is associated with conservation of higher-form charges that count the number of topological defects. Our formalism predicts, from first principles, the presence of propagating shear waves above a critical wave vector in liquids, thus giving a formal derivation of the phenomenon in terms of fundamental symmetries. The same picture provides also a theoretical explanation of the corresponding "positive sound dispersion" phenomenon for longitudinal sound. Importantly, accordingly to our theory, the main collective relaxation timescale of a liquid or a glass (known as the α relaxation for the latter) is given by the phase relaxation time, which is not necessarily related to the Maxwell time. Finally, we build a nonequilibrium effective action using the in-in formalism defined on the Schwinger-Keldysh contour, that further supports the emerging picture. In summary, our work suggests that the fundamental difference between solids, fluids, and glasses has to be identified with the associated generalized higher-form global symmetries and their topological structure, and that the Burgers vector for the displacement fields serves as a suitable topological order parameter distinguishing the solid (ordered) phase and the amorphous ones (fluids, glasses).