Plastically deforming crystals exhibit scale-free fluctuations that are similar to those observed in driven disordered elastic systems close to depinning, but the nature of the yielding critical point is still debated. Here, we study the marginal stability of ensembles of dislocations and compute their excitation spectrum in two and three dimensions. Our results show the presence of a singularity in the distribution of excitation stresses, i.e., the stress needed to make a localized region unstable, that is remarkably similar to the one measured in amorphous plasticity and spin glasses. These results allow us to understand recent observations of extended criticality in bursty crystal plasticity and explain how they originate from the presence of a pseudogap in the excitation spectrum.