Electric vehicle (EV) loads are playing an increasingly important role in improving the flexibility of power grid operation. The prerequisite for EV loads to participate in demand response (DR) is that the DR regulation strategy and corresponding DR potential must be accurately analyzed and evaluated. However, due to the uncertainty and differences in travel and charging behavior, DR potentials of EVs exhibit randomness and differ in time and space. In addition, it is difficult to obtain refined travel data and charging load data of large-scale EVs. Accordingly, this paper focuses on how to consider the various influencing factors of potential, and realize the quantitative evaluation of DR time-varying potential of an EV group based on small sample data. First, a travel activity model of the EV is established. Based on the actual travel data, the probability distributions of the key parameters of the travel model are obtained by kernel density estimation and probability statistical fitting. Then, combined with the charging behavior model, and based on Monte Carlo simulation, the load curve of the EV in a residential area is predicted. Considering the travel need of the EV, the peak-shaving potential, vehicle-to-grid discharge potential, and valley-filling potential of the EV under different DR strategies are calculated and analyzed, and the time-varying characteristics of the potential are analyzed. Finally, a case study is carried out with the actual data. The results show that the DR time-varying potential under different time periods and control strategies can be effectively evaluated. The maximum peak-shaving potential of 1000 EV aggregates is 2.7 MW, and the minimum is 0.25 MW. The maximum valley-filling potential is 2.1 MW, and the minimum is 0.3 MW. The research results can provide effective guidance for EVs to participate in day-ahead scheduling, and for the screening of target EVs.