The new incidence rate of rectal cancer accounts for 12.2% of all cancers, which seriously affects the health of the people. In addition, the early clinical diagnosis of T staging of rectal cancer will strongly support the formulation of effective treatment plans. However, it is difficult for clinicians to accurately determine the T stage of rectal cancer before operation because the site of rectal tumor invasion is not obvious. Therefore, this paper first proposes a bilinear feature fusion mechanism, which effectively avoids the problem of information loss in the process of convolution neural network training of rectal cancer MRI images; Secondly, the new weighted loss function designed can solve the problem of multi-example imbalance; Finally, the experiment proves that the constructed intelligent prediction strategy for Tstage of rectal cancer has a good accuracy, which provides a good auxiliary result for the clinical treatment of rectal cancer.