Weakly rigid drilling systems, such as the industrial robot, are widely used in aerospace, military, and other fields due to its good flexibility and large scope of operation. However, the weak rigidity can easily cause burrs, seriously affecting the precision of parts and product performance. To reduce the heavy deburring process and to improve continuous production and sustainable processing capacity, accurate prediction of burr quality is a prerequisite. Traditional burr forming theory cannot accurately predict the drilling defects. Data-driven approaches can be independent of prior knowledge and discover relationships between process parameters and machining precision directly from the data structure itself. Therefore, to take advantage of both approaches, a fusion model was established for burr classified prediction. On the one hand, the drilling and burr forming process was firstly modeled, and preliminary classification results for burrs were calculated. On the other hand, according to the measured data, the errors between initial calculation results and actual classification results were obtained and selected as the tag values of dataset, which served as inputs for the error compensation model of burrs. Finally, by training the network of TCN–DNN using the drilling data, the burr classified prediction in a weak rigid hole-making system was realized. Experimental results showed that compared with traditional drilling theory, the prediction accuracy of the proposed model improved by 25%, reaching 91.67%. The results can provide a basis for judging the process of burr post-treatment, which has practical guiding significance. This method is beneficial to reduce the heavy deburring process and to improve sustainable processing capacity.