PurposeThis bi-institutional study aimed to establish a robust model for predicting clinically significant prostate cancer (csPCa) (pathological grade group ≥ 2) in PI-RADS 3 lesions in the transition zone by comparing the performance of combination models.Materials and methodsThis study included 243 consecutive men who underwent 3-Tesla magnetic resonance imaging (MRI) and ultrasound-guided transrectal biopsy from January 2020 and April 2022 which is divided into a training cohort of 170 patients and a separate testing cohort of 73 patients. T2WI and DWI images were manually segmented for PI-RADS 3 lesions for the mean ADC and radiomic analysis. Predictive clinical factors were identified using both univariate and multivariate logistic models. The least absolute shrinkage and selection operator (LASSO) regression models were deployed for feature selection and for constructing radiomic signatures. We developed nine models utilizing clinical factors, radiological features, and radiomics, leveraging logistic and XGboost methods. The performances of these models was subsequently compared using Receiver Operating Characteristic (ROC) analysis and the Delong test.ResultsOut of the 243 participants with a median age of 70 years, 30 were diagnosed with csPCa, leaving 213 without a csPCa diagnosis. Prostate-specific antigen density (PSAD) stood out as the only significant clinical factor (odds ratio [OR], 1.068; 95% confidence interval [CI], 1.029–1.115), discovered through the univariate and multivariate logistic models. Seven radiomic features correlated with csPCa prediction. Notably, the XGboost model outperformed eight other models (AUC of the training cohort: 0.949, and validation cohort: 0.913). However, it did not surpass the PSAD+MADC model (P > 0.05) in the training and testing cohorts (AUC, 0.949 vs. 0.888 and 0.913 vs. 0.854, respectively).ConclusionThe machine learning XGboost model presented the best performance in predicting csPCa in PI-RADS 3 lesions within the transitional zone. However, the addition of radiomic classifiers did not display any significant enhancement over the compound model of clinical and radiological findings. The most exemplary and generalized option for quantitative prostate evaluation was Mean ADC+PSAD.