We present a microscopic theory of nonlinear damping and dephasing of low-frequency eigenmodes in nanomechanical and micromechanical systems. The mechanism of the both effects is scattering of thermally excited vibrational modes off the considered eigenmode. The scattering is accompanied by energy transfer of 2 ω 0 for nonlinear damping and is quasielastic for dephasing. We develop a formalism that allows studying both spatially uniform systems and systems with a strong nonuniformity, which is smooth on the typical wavelength of thermal modes but is pronounced on their mean free path. The formalism accounts for the decay of thermal modes, which plays a major role in the nonlinear damping and dephasing. We identify the nonlinear analogs of the Landau-Rumer, thermoelastic, and Akhiezer mechanisms and find the dependence of the relaxation parameters on the temperature and the geometry of a system.