We report on experiments addressing the non-linear interaction between a nano-mechanical mode and position fluctuations. The Duffing non-linearity transduces the Brownian motion of the mode, and of other non-linearly coupled ones, into frequency noise. This mechanism, ubiquitous to all weaklynonlinear resonators thermalized to a bath, results in a phase diffusion process altering the motion: two limit behaviors appear, analogous to motional narrowing and inhomogeneous broadening in NMR. Their crossover is found to depend non-trivially on the ratio of the frequency noise correlation time to its magnitude. Our measurements obtained over an unprecedented range covering the two limits match the theory of Y. Zhang and M. I. Dykman, Phys. Rev. B 92, 165419 (2015), with no free parameters. We finally discuss the fundamental bound on frequency resolution set by this mechanism, which is not marginal for bottom-up nanostructures.arXiv:1704.06119v2 [cond-mat.mes-hall]