Theoretical models of ecological specialization commonly assume that adaptation to one environment leads to fitness reductions (costs) in others. However, empirical studies often fail to detect such costs. We addressed this conundrum using experimental evolution with Escherichia coli in several homogeneous and heterogeneous environments at multiple population sizes. We found that in heterogeneous environments, smaller populations paid significant costs, but larger ones avoided them altogether. Contrastingly, in homogeneous environments, larger populations paid more costs than the smaller ones. Overall, large population sizes and heterogeneous environments led to cost avoidance when present together but not on their own. Whole-genome whole-population sequencing revealed that the enrichment of multiple mutations within the same lineage (and not subdivision into multiple distinct specialist subpopulations) was the mechanism of cost avoidance. Since the conditions revealed by our study for avoiding costs are widespread, it explains why the costs expected in theory are rarely detected in experiments.