Following experimental evidence that vibrational polaritons, formed from collective vibrational strong coupling (VSC) in optical microcavities, can modify ground-state reaction rates, a spate of theoretical explanations relying on cavity-induced frictions have been proposed through the Pollak−Grabert−Hanggi (PGH) theory, which goes beyond transition state theory (TST). However, by considering only a single reacting molecule coupled to light, these works do not capture the ensemble effects present in experiments. Moreover, the relevant light−matter coupling should have been √N times smaller than those used by preceding works, where N ≈ 10 6 to 10 12 is the ensemble size. In this work, we explain why this distinction is significant and can nullify effects from these cavity-induced frictions. By analytically extending the cavity PGH model to realistic values of N, we show how this model succumbs to the polariton "large N problem", that is, the situation whereby the single reacting molecule feels only a tiny 1/N part of the collective light−matter interaction intensity, where N is large.