The early motivation for and development of diagonal increments to ease matrix inversion in least squares (LS) problems is discussed. It is noted that this diagonal incrementation evolved from three major directions: modification of existing methodology in non-linear LS, utilization of additional information in linear regression, and the improvement of the numerical condition of a matrix. The interplay among these factors, and the advent of ridge regression are considered in an historical and comparative framework.