False positives on multiparametric MRIs (mp-MRIs) result in many unnecessary invasive biopsies in men with clinically insignificant diseases. This study investigated whether quantitative diffusion MRI could differentiate between false positives, true positives and normal tissue non-invasively. Thirty-eight patients underwent mp-MRI and Vascular, Extracellular and Restricted Diffusion for Cytometry in Tumors (VERDICT) MRI, followed by transperineal biopsy. The patients were categorized into two groups following biopsy: (1) significant cancer—true positive, 19 patients; (2) atrophy/inflammation/high-grade prostatic intraepithelial neoplasia (PIN)—false positive, 19 patients. The clinical apparent diffusion coefficient (ADC) values were obtained, and the intravoxel incoherent motion (IVIM), diffusion kurtosis imaging (DKI) and VERDICT models were fitted via deep learning. Significant differences (p < 0.05) between true positive and false positive lesions were found in ADC, IVIM perfusion fraction (f) and diffusivity (D), DKI diffusivity (DK) (p < 0.0001) and kurtosis (K) and VERDICT intracellular volume fraction (fIC), extracellular–extravascular volume fraction (fEES) and diffusivity (dEES) values. Significant differences between false positives and normal tissue were found for the VERDICT fIC (p = 0.004) and IVIM D. These results demonstrate that model-based diffusion MRI could reduce unnecessary biopsies occurring due to false positive prostate lesions and shows promising sensitivity to benign diseases.