Temporal Kerr cavity solitons are pulses of light that can persist in coherently-driven, dispersive resonators with Kerr-type nonlinearity. Many studies have shown that such solitons can react to parameter inhomogeneities (e.g. variations in the complex amplitude of the driving field) by experiencing a temporal drift. The vast majority of such studies assume that the inhomogeneity varies slowly across the soliton, leading to the prediction that the soliton drift rate is linearly proportional to the gradient of the inhomogeneity at the soliton position. However, the assumption of a slowly-varying inhomogeneity may not hold true under all situations, e.g. when using bichromatic driving or in the presence of third-order dispersion that gives rise to an extended dispersive wave tail. Here we report on theoretical and numerical results pertaining to the behavior of dissipative temporal Kerr cavity solitons under conditions where parameter inhomogeneities vary nonlinearly across the width of the soliton. In this case, the soliton velocity is dictated by the full overlap between its so-called adjoint neutral mode and the parameter perturbation, which we show can yield dynamics that are manifestly at odds with the common wisdom of motion dependent solely upon the gradient of the inhomogeneity. We also investigate how the presence of third-order dispersion and the associated dispersive wave tail changes the motion induced by parameter inhomogeneities. We find that the dispersive wave tail as such does not contribute to the soliton motion; instead, higher-order dispersion yields counter-intuitive influences on the soliton motion. Our results provide new insights to the behavior of temporal cavity solitons in the presence of parameter inhomogeneities, and can impact systems employing pulsed or bichromatic pumping and/or resonators with non-negligible higherorder dispersion.