We test a hypothesis for the origin of dynamical heterogeneity in slowly relaxing systems, namely that it emerges from soft (Goldstone) modes associated with a broken continuous symmetry under time reparametrizations. We do this by constructing coarse grained observables and decomposing the fluctuations of these observables into transverse components, which are associated with the postulated time-fluctuation soft modes, and a longitudinal component, which represents the rest of the fluctuations. Our test is performed on data obtained in simulations of four models of structural glasses. As the hypothesis predicts, we find that the time reparametrization fluctuations become increasingly dominant as temperature is lowered and timescales are increased. More specifically, the ratio between the strengths of the transverse fluctuations and the longitudinal fluctuations grows as a function of the dynamical susceptibility, χ4, which represents the strength of the dynamical heterogeneity; and the correlation volumes for the transverse fluctuations are approximately proportional to those for the dynamical heterogeneity, while the correlation volumes for the longitudinal fluctuations remain small and approximately constant.