Diagnostic histopathology is facing increasing demands due to aging populations and expanding healthcare programs. Semi-automated diagnostic systems employing deep learning methods are one approach to alleviate this pressure, with promising results for many routine diagnostic procedures. However, one major issue with deep learning approaches is their lack of interpretability - after adequate training they perform their assigned tasks admirably, but do not explain how they reach their conclusions. Knowledge of how a given method performs its task with high sensitivity and specificity would be advantageous to understand the key features responsible for diagnosis, and should in turn allow fine-tuning of deep learning approaches. This paper presents a deep learning-based system for carcinoma detection in whole slide images of prostate core biopsies, achieving state-of-the-art performance; 100% area under curve and sensitivity of 0.978 for 8 detected false positives on average per slide. Furthermore, we investigated various methods to extract the key features used by the neural network for classification. Of these, the technique called occlusion, adapted to whole slide images, analyzes the sensitivity of the detection system to changes in the input images. This technique produces heatmaps indicating which parts of the image have the strongest impact on the system's output that a histopathologist can examine to identify the network's reasoning behind a given classification. Reassuringly, the heatmaps identified several prevailing histomorphological features characterizing carcinoma, e.g. single-layered epithelium, presence of small lumina, and hyperchromatic nuclei with halos. A convincing finding was the recognition of their mimickers in non-neoplastic tissue. The results show that the neural network approach to recognize prostatic cancer is similar to that taken by a human pathologist at medium optical resolution. The use of explainability heatmaps provides added value for automated digital pathology to analyze and fine-tune deep learning systems, and improves trust in computer-based decisions.