Semiconductor quantum dots have recently emerged as a leading platform to efficiently generate highly indistinguishable photons, and this work addresses the timely question of how good these solid-state sources can ultimately be. We establish the crucial role of lattice relaxation in these systems in giving rise to trade-offs between indistinguishability and efficiency. We analyse the two source architectures most commonly employed: a quantum dot embedded in a waveguide and a quantum dot coupled to an optical cavity. For waveguides, we demonstrate that the broadband Purcell effect results in a simple inverse relationship, where indistinguishability and efficiency cannot be simultaneously increased. For cavities, the frequency selectivity of the Purcell enhancement results in a more subtle trade-off, where indistinguishability and efficiency can be simultaneously increased, though by the same mechanism not arbitrarily, limiting a source with near-unity indistinguishability (> 99%) to an efficiency of approximately 96% for realistic parameters.The efficient generation of on-demand highly indistinguishable photons remains a barrier to the scalability of a number of photonic quantum technologies 1-4 . To this end, attention has recently turned towards solid-state systems, and in particular semiconductor quantum dots (QDs) 5-13 , which can not only emit a single photon with high quantum efficiency, but can be easily integrated into larger photonic structures 14 , resulting in photons being emitted into a well-defined mode and direction. Highly directional emission is crucial to the overall efficiency of the source, and is typically achieved by either placing the QD in a waveguide with low out-of-plane scattering 15,16 , or by coupling resonantly to an optical cavity mode 6-9,12,13 . Nevertheless, the solid-state nature of QDs leads to strong coupling between the electronic degrees of freedom and their local environment; fluctuating charges 17 , nuclear spins 18,19 , and lattice vibrations 20-23 all lead to a suppression of photon coherence and a resulting reduction in indistinguishability 11,[24][25][26][27] . While early experiments were indeed limited by these factors 6-9 , improvements in fabrication and resonant excitation techniques have steadily increased photon indistinguishability to levels now exceeding 99% in resonantly coupled QD-cavity systems 12,13 . Photon extraction efficiencies have also steady improved, with the highest values reaching 98% in a photonic crystal waveguide 16 .Despite this impressive progress, a system boasting very high (> 99%) indistinguishability and efficiency as required for e.g. cluster state quantum computing 28 remains elusive. Strategies aimed at achieving such a source typically focus on engineering the photonic environment in order to maximise the Purcell effect 29,30 , where the QD emission rate becomes F P Γ, with Γ the bulk emission rate and F P the Purcell factor 29 . Modelling a QD as a simple two-level-system with a Markovian phenomenological dephasing rate γ, the Purc...