We investigate the cosmological perturbations in the context of a Scalar-Tensor-Vector theory of Gravity known as MOG in the literature. Although MOG is plagued by ghosts, it does not suffer from tachyonic instability. Therefore it can be considered as a valid theory, at least in the classical limit. In the weak field limit, MOG increases the strength of gravity, and consequently addresses some astrophysical tests, without invoking dark matter particles. Recent investigations show that MOG reproduces a viable background cosmological evolution comparable to ΛCDM. However, the matter dominated era is slightly faster in MOG compared to ΛCDM. In this paper, in order to clarify whether this evolution is compatible with observations, we study the linear matter perturbations and estimate the relevant modified gravity parameters. In contrast to some current claims in the relevant literature, we show that MOG reduces the growth rate of the perturbations. We then compare MOG to the redshift space distortion (RSD) data. We find that MOG yields a much higher value than ΛCDM for the power spectrum parameter σ 8 . Although MOG cannot yet be ruled out by RSD data alone, the low growth and high σ 8 constitute a powerful challenge to the cosmological viability of MOG.