In this paper, we mainly use random forest and broad learning system (BLS) to predict rectal cancer. A total of 246 participants with computed tomography (CT) image records were enrolled. The total model in the training set (combined with imaging and clinical indicators) has the best prediction result, with the area under the curve (AUC) of 0.999 (95% confidence internal (CI): 0.996–1.000) and the accuracy of 0.990 (95%CI: 0.976–1.000). Model 3, the general model in the test set, has the best prediction result, with the AUC of 0.962 (95%CI: 0.915–1.000) and the accuracy of 0.920 (95%CI: 0.845–0.995). The results of the model using random forest prediction are compared with those using BLS prediction. It can be found that there is no statistical difference between the two results. Our prediction model combined with image features has a good prediction result, and this image feature is the most important among all features. Consequently, we can successfully predict rectal cancer through a combination of the clinical indicators and the comprehensive indicators of CT image characteristics in four different periods (plain scan, vein, artery, and excretion).