Determining the structure of gene regulatory networks (GRNs) is a central problem in biology, with a variety of inference methods available for different types of data. However, for a prominent and intricate scenario with single-cell gene expression data collected post-intervention across multiple time points, where joint distributions remain unknown, there is only one known specifically developed method, which does not fully utilize the rich information contained in this data type. In response, we introduce an inference approach tailored to this challenging context: netWork infErence by covariaNce DYnamics, dubbed WENDY. The core idea of WENDY is to model the dynamics of the covariance matrix, and solve this dynamics as an optimization problem to determine the regulatory relationships. To assess its efficacy, we benchmark WENDY against alternative inference methods using synthetic data. Our findings underscore WENDY's robust performance across diverse synthetic datasets. Moreover, we deploy WENDY to analyze three distinct experimental datasets, uncovering potential gene regulatory mechanisms.