The short‐term origin‐destination (OD) forecast in bus rapid transit (BRT) can provide future OD demand, which means that transport plan personnel can take measures in advance, such as adjusting the departure schedule, to avoid congestion. This emphasizes the importance of accurate and stable OD forecasting in real‐time. Deep learning methods have been widely applied due to their superior performance in capturing dynamic spatiotemporal and nonlinear features. However, current methods based on deep learning and data‐driven cannot simultaneously address the problems of data availability and sparsity, and their ability to capture dynamic spatiotemporal patterns is insufficient, which cannot provide satisfactory accuracy and stability. To address these problems, this paper proposes a method for forecasting the OD demand in BRT based on dual attention multi‐scale convolutional network (DAMSCN). Considering the three temporal properties of OD, that is, closeness, period and trend dependencies. DAMSCN uses multi‐scale convolution (MSC), dual attention mechanism (DAt) and stacked convolutional long short‐term memory networks (ConvLSTM) to model the three kinds of properties. The MSC module converts sparse OD information into dense useful features to solve the problem of data sparsity. The DAt and ConvLSTM module model the dependencies relationship in the spatial and temporal dimensions, which can improve the spatiotemporal features extraction ability. DAMSCN also combine inflow information to enhance data availability. The three properties are dynamically aggregated and further integrated with external influences (e.g. weather conditions etc.) to obtain the final forecasting result. The accuracy and stability are evaluated on the Xiamen BRT (XMBRT) and Shanghai Metro datasets (SHMetro). The experimental results show that the DAMSCN is superior to other method. Compared to the best baseline, on the XMBRT, MAE decreased by 8.60%, and RMSE decreased by 4.81%. On the SHMetro, MAE decreased by 7.22%, and RMSE decreased by 8.96%.