The time domain expression of the lightning electromagnetic fields (LEMF) contains multiple integration processes, and the direct solution tends to take up a lot of calculation time. In this paper, the structure of the time domain expression of the LEMF is analyzed, the idea that using the matrix pencil method to realize the fast evaluation of the LEMF is proposed, and the feasibility of this method for calculating the LEMF is discussed. Besides, several numerical examples are given to prove the efficiency and correctness of the proposed method when the parameters of the observation point and ground are varied.