State estimation is a crucial problem in modern industries and has been widely applied across various fields. The performance of the estimator depends on the quality of the measurement data. Measurements being corrupted by outliers is becoming an unavoidable phenomenon that leads to degradation of estimator performance. It is critical to develop estimators with outlier suppression capabilities to mitigate the adverse impact of measurement outliers. In this paper, we propose an effective outlier suppression technique for discrete-time linear systems within the framework of moving horizon estimation (MHE). The proposed estimator solves the issues of poor estimation accuracy and low computational efficiency among the existing MHE-based outlier-robust estimators. Moreover, the proposed method allows us to not only achieve robust state estimation but also detect outliers. Specifically, we propose a set of least-squares cost functions and an outlier identification mechanism to implement the estimation process. Subsequently, the stability of the estimation error of the proposed estimator is demonstrated. The estimation error can achieve exponential convergence by choosing appropriate design parameters. Lastly, the proposed estimator is applied to target tracking simulations and compared with state-of-the-art outlier-robust estimation methods, confirming the effectiveness and superiority of the proposed estimator.