Deformation prediction is the basis of deformation control in manufacturing process planning. This paper presents an on-line part deformation prediction method using a deep learning model during numerical control machining process, which is different from traditional methods based on finite element simulation of stress release prior to the actual machining process. A fourth-order tensor model is proposed to represent the continuous part geometric information, process information, and monitoring information, which is used as the input to the deep learning model. A deep learning framework with a Conventional Neural Network and a Recurrent Neural Network has been constructed and trained by monitored deformation data and process information associated with interim part geometric information. The proposed method can be generalised for different parts with certain similarities and has the potential to provide a reference for an adaptive machining control strategy for reducing part deformation. The proposed method was validated by actual machining experiments, and the results show that the prediction accuracy has been improved compared with existing methods. Furthermore, this paper shifts the difficult problem of residual stress measurement and off-line deformation prediction to the solution of on-line deformation prediction based on deformation monitoring data.