Populations threatened by an abrupt environmental change—due to rapid climate change, pathogens or invasive competitors—may survive if they possess or generate genetic combinations adapted to the novel, challenging condition. If these genotypes are initially rare or non-existent, the emergence of lineages that allow a declining population to survive is known as ‘evolutionary rescue’. By contrast, the genotypes required for survival could, by chance, be common before the environmental change. Here, considering both of these possibilities, we find that the risk of extinction can be lower in very small or very large populations, but peaks at intermediate population sizes. This pattern occurs when the survival genotype has a small deleterious effect before the environmental change. Since mildly deleterious mutations constitute a large fraction of empirically measured fitness effects, we suggest that this unexpected result—an intermediate size that puts a population at a greater risk of extinction—may not be unusual in the face of environmental change.