Predicting city-scale taxi origin-destination (OD) flows takes an important role in understanding passengers' travel demands as well as managing taxi operation and scheduling. But the complex spatial dependencies and temporal dynamics make this problem challenging. In this paper, a hybrid deep neural network prediction model based on convolutional LSTM (ConvLSTM) is proposed. For improving the prediction accuracy, the implicit correlation between travel time and OD flow is explored and they are combined as inputs of the prediction model. Moreover, in order to realize OD flows prediction at the road network level, and solve the problem that grid-based representation method cannot distinguish traffic flow at different heights, such as in multi-layer overpass areas, this paper presents a grid and road nested method to represent ODs. With the time of day partitioned into time slots, OD flows are extracted and predicted in both spatial and temporal domain. In the experiment, real taxi data are used to verify the proposed model and prediction method fully. And the experimental results show that the proposed model can effectively predict city-wide taxi OD flow, and outperforms the typical time sequence models and existing deep neural network models. INDEX TERMS Prediction of dynamic origin-destination flows, hybrid deep neural network, demands for taxis, GPS trajectory of taxis.