The Prostate Imaging and Reporting Data System (PI-RADS) has a key role in the management of prostate cancer (PCa). However, the clinical interpretation of PI-RADS 3 score lesions may be challenging and misleading, thus postponing PCa diagnosis to biopsy outcome. Multiparametric magnetic resonance imaging (mpMRI) radiomic analysis may represent a stand-alone noninvasive tool for PCa diagnosis. Hence, this study aims at developing a mpMRI-based radiomic PCa diagnostic model in a cohort of PI-RADS 3 lesions. We enrolled 133 patients with 155 PI-RADS 3 lesions, 84 of which had PCa confirmation by fusion biopsy. Local radiomic features were generated from apparent diffusion coefficient maps, and the four most informative were selected using LASSO, the Wilcoxon rank-sum test (p < 0.001), and support vector machines (SVMs). The selected features where augmented and used to train an SVM classifier, externally validated on a holdout subset. Linear and second-order polynomial kernels were exploited, and their predictive performance compared through receiver operating characteristics (ROC)-related metrics. On the test set, the highest performance, equally for both kernels, was specificity = 76%, sensitivity = 78%, positive predictive value = 80%, and negative predictive value = 74%. Our findings substantially improve radiologist interpretation of PI-RADS 3 lesions and let us advance towards an image-driven PCa diagnosis.