Reconstructing large-scale gene regulatory networks (GRNs) is a challenging problem in the field of computational biology. Various methods for inferring GRNs have been developed, but they fail to accurately infer GRNs with a large number of genes. Additionally, the existing evaluation indexes for evaluating the constructed networks have obvious disadvantages because GRNs in most biological systems are sparse. In this paper, we develop a new method for inferring GRNs based on randomized singular value decomposition (RSVD) and ordinary differential equation (ODE)-based optimization, denoted as IGRSVD, from large-scale time series data with noise. The three major contributions of this paper are as follows. First, the IGRSVD algorithm uses the RSVD to handle the noise and reduce the original large-scale data into small-scale problems. Second, we propose two new evaluated indexes, the expected value accuracy (EVA) and the expected value error (EVE), to evaluate the performance of inferred networks by considering the sparse features in the network. Finally, the proposed IGRSVD algorithm is compared with the existing SVD algorithm and PCA_CMI algorithm using four subsets from E. coli and datasets from DREAM challenge. The experimental results demonstrate that the IGRSVD algorithm is effective and more suitable for reconstructing large-scale networks.