Y and W chromosomes often stop recombining and degenerate. Different theories have been proposed to explain these observations. The classic theory supposes that recombination arrest is favored because it allows to permanently link sex-antagonistic alleles to the sex in which they are favorable. The regulatory theory supposes that recombination arrest is initially caused by lucky inversions on the Y or W carrying fewer deleterious mutations than average, which then become selectively stabilized by the evolution of nascent dosage compensation creating sex-antagonistic regulatory effects. Last, the shelter theory also supposes that lucky Y (or W) inversions do occur, but that they are maintained in the long term because of a lack of genetic variation to restore recombination. The last two theories are comparable in the sense that they do not require sexually dimorphic traits, and are based primarily on the flux of deleterious mutations. In this paper, we focus on the shelter theory and show that contrarily to previous claims, it can hardly explain the evolution of sex chromosomes. We show that it is not robust to extremely low rates of recombination restoration, that it would cause population extinction rather than Y degeneration, and that it is orders of magnitudes less efficient than the mechanism described in the regulatory theory. We also argue that the assumption of a lack of genetic variation to restore recombination is unplausible given known mechanisms of recombination variation, and given the extremely small rates of restoration that are sufficient to make the shelter theory inoperant.