Patient-derived organoids (PDOs) offer new opportunities to model various cancers. However, their application in prostate cancer (PCa) has been hampered by poor success rates and overgrowth of cell types which are not representative of the patient samples. By exploiting a cohort of 164 PCa patient samples and tuning several culture parameters, we show that an extracellular matrix-free (ECM)-free culture system increases the take-rate of PDOs with luminal-like and PCa features. Single-cell RNA sequencing (scRNA-seq) reveals that ECM-free PDOs comprise cell populations associated with known PCa signatures and exhibit transcriptomic resemblance with their respective parental tumors. In addition, we define organoid-associated cell type signatures and identify markers discriminating tumors versus benign cells ex vivo and in situ. Furthermore, we generate the first prostate PDO single-cell atlas integrating previously-published scRNA-seq datasets and our newly-generated data. We show that Matrigel-based organoid cultures derived from primary PCa are essentially composed of benign-like epithelial cells, irrespective of the dataset or the malignant nature of the tissue of origin. In contrast, ECM-free conditions maintain heterogenous patient-specific luminal tumor cell populations and enrich in intermediate cell types. Ultimately, our work will significantly enhance the potential of PDOs in basic and translational PCa research.