The accurate prediction of passenger flow is crucial in improving the quality of the service of intercity high-speed railways. At present, there are a few studies on such predictions for railway origin–destination (O-D) pairs, and usually only a single factor is considered, yielding a low prediction accuracy. In this paper, we propose a neural network model based on multi-source data (NN-MSD) to predict the O-D passenger flow of intercity high-speed railways at different times in one day in the short term, considering the factors of time, space, and weather. Firstly, the factors that influence time-varying passenger flow are analyzed based on multi-source data. The cyclical characteristics, spatial and temporal fusion characteristics, and weather characteristics are extracted. Secondly, a neural network model including three modules is designed based on the characteristics. A fully connected network (FCN) model is used in the first module to process the classification data. A bi-directional Long Short-Term Memory (Bi-LSTM) model is used in the second module to process the time series data. The results of the first module and the second module are spliced and fused in the third module using an FCN model. Finally, an experimental analysis is performed for the Guangzhou–Zhuhai intercity high-speed railway in China, in which three groups of comparison experiments are designed. The results show that the proposed NN-MSD model can predict many O-D pairs with a high and stable accuracy, which outperforms the baseline models, and multi-source data are very helpful in improving the prediction accuracy.