Anomaly Detection (AD) in images is a fundamental computer vision problem and refers to identifying images that deviate significantly from normality. State-of-the-art AD algorithms commonly learn a model of normality from scratch using task specific datasets in either semi-supervised or selfsupervised manner. We follow an alternative approach, and model the distribution of normal data in deep feature representations learned from ImageNet via a multivariate Gaussian (MVG). This lightweight approach achieves a new state of the art in AD on the public MVTec AD dataset. In addition to the empirical benefits, we give a clear motivation for the seemingly simplistic approach via the ties between deep generative and discriminative modeling revealed recently. We further elucidate why ImageNet representations are discriminative in the transfer learning AD setting using Principal Component Analysis. Here, we find that the principal components containing little variance in normal data are the ones crucial for discriminating between normal and anomalous instances, giving an explanation for the unreasonable effectiveness of our approach. We also investigate setting the working point of our approach by selecting acceptable False Positive Rate thresholds based on the MVG assumption as well as the resistance of our approach to unlabeled anomalies in the dataset. Finally, we investigate whether our approach is prone to exploiting spurious correlations using explainable AI techniques. Code is publicly available at https://github.com/ORippler/gaussian-ad-mvtec.