This paper explores a linear state estimation problem in non-Gaussian setting and suggests a computationally simple estimator based on the maximum correntropy criterion Kalman filter (MCC-KF). The first MCC-KF method was developed in Joseph stabilized form. It requires two n × n and one m × m matrix inversions, where n is a dimension of unknown dynamic state to be estimated, and m is a dimension of available measurement vector. Therefore, the estimator becomes impractical when the system dimensions increase. Our previous work has suggested an improved MCC-KF estimator (IMCC-KF) and its factored-from (square-root) implementations that enhance the MCC-KF estimation quality and numerical robustness against roundoff errors. However, the proposed IMCC-KF and its square-root implementations still require the m × m matrix inversion in each iteration step of the filter. For numerical stability and computational complexity reasons it is preferable to avoid the matrix inversion operation. In this paper, we suggest a new IMCC-KF algorithm that is more accurate and computationally cheaper than the original MCC-KF and previously suggested IMCC-KF. Furthermore, compared with stable square-root algorithms, the new method is also accurate, but less computationally expensive. The results of numerical experiments substantiate the mentioned properties of the new estimator on numerical examples.