Spatio‐temporal modeling is an active research problem with broad applications. In this problem, proper description and estimation of the data covariance structure plays an important role. In the literature, most available methods assume that the data covariance is stationary and follows a specific parametric form. In practice, however, such assumptions are hardly valid or difficult to verify. In this paper, we propose a new and flexible method for estimating the underlying covariance structure. Our proposed method does not require the covariance to be stationary or follow a parametric form. It can accommodate nonparametric space‐time–varying mean structure of the observed data. Under some mild regularity conditions, it is shown that our estimated covariance structure converges to the true covariance structure. The proposed method is also justified numerically by a simulation study and an application to a hand, foot, and mouth disease data.