In the context of regression analysis, we propose an estimation method capable of producing estimators that are closer to the true parameters than standard estimators when the residuals are non-normally distributed and when outliers are present. We achieve this improvement by minimizing the norm of the errors in general L<sup>p</sup> spaces, as opposed to minimizing the norm of the errors in the typical L<sup>2</sup> space, corresponding to Ordinary Least Squares (OLS). The generalized model proposed here—the Ordinary Least Powers (OLP) model—can implicitly adjust its sensitivity to outliers by changing its parameter <i>p</i>, the exponent of the absolute value of the residuals. Especially for residuals of large magnitude, such as those stemming from outliers or heavy-tailed distributions, different values of <i>p</i> will implicitly exert different relative weights on the corresponding residual observation. We fitted OLS and OLP models on simulated data under varying distributions providing outlying observations and compared the mean squared errors relative to the true parameters. We found that OLP models with smaller <i>p</i>'s produce estimators closer to the true parameters when the probability distribution of the error term is exponential or Cauchy, and larger <i>p</i>'s produce closer estimators to the true parameters when the error terms are distributed uniformly.