Deep learning (DL)-based PV Power Forecasting (PVPF) emerged nowadays as a promising research direction to intelligentize energy systems. With the massive smart meter integration, DL takes advantage of the large-scale and multi-source data representations to achieve a spectacular performance and high PV forecastability potential compared to classical models. This review article taxonomically dives into the nitty-gritty of the mainstream DL-based PVPF methods while showcasing their strengths and weaknesses. Firstly, we draw connections between PVPF and DL approaches and show how this relation might cross-fertilize or extend both directions. Then, fruitful discussions are conducted based on three classes: discriminative learning, generative learning, and deep reinforcement learning. In addition, this review analyzes recent automatic architecture optimization algorithms for DL-based PVPF. Next, the notable DL technologies are thoroughly described. These technologies include federated learning, deep transfer learning, incremental learning, and big data DL. After that, DL methods are taxonomized into deterministic and probabilistic PVPF. Finally, this review concludes with some research gaps and hints about future challenges and research directions in driving the further success of DL techniques to PVPF applications. By compiling this study, we expect to help aspiring stakeholders widen their knowledge of the staggering potential of DL for PVPF.