Black Holes are possibly the most enigmatic objects in our Universe. From their detection in gravitational waves upon their mergers, to their snapshot eating at the centres of galaxies, black hole astrophysics has undergone an observational renaissance in the past 4 years. Nevertheless, they remain active playgrounds for strong gravity and quantum effects, where novel aspects of the elusive theory of quantum gravity may be hard at work. In this review article, we provide an overview of the strong motivations for why "Quantum Black Holes" may be radically different from their classical counterparts in Einstein's General Relativity. We then discuss the observational signatures of quantum black holes, focusing on gravitational wave echoes as smoking guns for quantum horizons (or exotic compact objects), which have led to significant recent excitement and activity. We review the theoretical underpinning of gravitational wave echoes and critically examine the seemingly contradictory observational claims regarding their (non-)existence. Finally, we discuss the future theoretical and observational landscape for unraveling the "Quantum Black Holes in the Sky".