The models of time-varying network have a profound impact on the study of virus spreading on the networks. On the basis of an activity-driven memory evolution model, a time-varying spatial memory model (TSM) is proposed. In the TSM model, the cumulative number of connections between nodes is recorded, and the spatiality of nodes is considered at the same time. Therefore, the active nodes tend to connect the nodes with high intimacy and close proximity. Then, the TSM model is applied to epidemic spreading, and the epidemic spreading on different models is compared. To verify the universality of the TSM model, this model is also applied to rumor spreading, and it is proved that it can also play a good inhibiting effect. We find that, in the TSM network, the introduction of spatiality and memory can slow down the propagation speed and narrow the propagation scope of disease or rumor, and memory is more important. We then explore the impact of different prevention and control methods on pandemic spreading to provide reference for COVID-19 management control and find when the activity of node is restricted, the spreading will be controlled. As floating population has been acknowledged as a key parameter that affects the situation of COVID-19 after work resumption, the factor of population mobility is introduced to calculate the interregional population interaction rate, and the time-varying interregional epidemic model is established. Finally, our results of infectious disease parameters based on daily cases are in good agreement with the real data, and the effectiveness of different control measures is evaluated.