The current uneven deployment of charging stations for electric vehicles (EVs) requires a reliable prediction solution for smart grids. Existing traffic prediction assumes that users’ charging durations are constant in a given period and may not be realistic. In fact, the actual charging duration is affected by various factors including battery status, user behavior, and environment factors, leading to significant differences in charging duration among different charging stations. Ignoring these facts would severely affect the prediction accuracy. In this paper, a Transformer-based prediction of user charging durations is proposed. Moreover, a data aggregation scheme with privacy protection is designed. Specifically, the Transformer charging duration prediction dynamically selects active and reliable temporal nodes through a truncated attention mechanism. This effectively eliminates abnormal fluctuations in prediction accuracy. The proposed data aggregation scheme employs a federated learning framework, which centrally trains the Transformer without any prior knowledge and achieves reliable data aggregation through a dynamic data flow convergence mechanism. Furthermore, by leveraging the statistical characteristics of model parameters, an effective model parameter updating method is investigated to reduce the communication bandwidth requirements of federated learning. Experimental results show that the proposed algorithm can achieve the novel prediction accuracy of charging durations as well as protect user data privacy.