Primordial Black Holes (PBHs) may form in the early Universe, from the gravitational
collapse of large density perturbations, generated by large quantum fluctuations during
inflation. Since PBHs form from rare over-densities, their abundance is sensitive to the tail of
the primordial probability distribution function (PDF) of the perturbations. It is therefore
important to calculate the full PDF of the perturbations, which can be done non-perturbatively
using the `stochastic inflation' framework. In single field inflation models generating large
enough perturbations to produce an interesting abundance of PBHs requires violation of slow
roll. It is therefore necessary to extend the stochastic inflation formalism beyond slow roll. A
crucial ingredient for this are the stochastic noise matrix elements of the inflaton potential. We
carry out analytical and numerical calculations of these matrix elements for a potential with a
feature which violates slow roll and produces large, potentially PBH generating, perturbations.
We find that the transition to an ultra slow-roll phase results in the momentum induced noise
terms becoming larger than the field noise whilst each of them falls exponentially for a few
e-folds. The noise terms then start rising with their original order restored, before approaching
constant values which depend on the nature of the slow roll parameters in the post transition
epoch. This will significantly impact the quantum diffusion of the coarse-grained inflaton field,
and hence the PDF of the perturbations and the PBH mass fraction.