Chronically elevated ungulate browse pressure in temperate forests worldwide often generates ecological legacies characterized by low plant diversity and contributes to the formation of dense, nearly monodominant, and highly recalcitrant layers of understory vegetation. Once established, these recalcitrant layers combined with continued browsing may jointly constrain tree establishment and diversity so completely that understory recovery may be unattainable without mitigating browse pressure, the recalcitrant layer, or both. Here, we investigate the independent and synergistic effects of both white‐tailed deer (Odocoileus virginianus) browsing and hay‐scented fern (Dennstaedtia punctilobula) competition on tree regeneration in a 10‐year experiment. Specifically, we examine how tree seedling establishment, growth, and composition are filtered by fern cover versus fern removal (gaps), browser presence versus absence (exclosures), and their combined effects during 10 years at three hardwood forest sites in Pennsylvania, USA. Fern gaps enhanced establishment for multiple tree species, increasing seedling density and diversity, particularly in the first 3 years post‐treatment, and enhancing richness (≤1 species) over the course of the experiment. Excluding deer for a decade increased the height growth of other regeneration and altered species composition, but had no effect on diversity, richness, and density. Notably, we observed higher Prunus serotina seedling densities outside exclosures, possibly due to greater secondary dispersal. We argue that browsing legacies in second growth forests established at the turn of the last century created two conditions inimical to diverse forest regeneration: an overstory dominated by two species, P. serotina and Acer rubrum (86% of basal area), and a dense recalcitrant understory layer dominated by a native fern. The first condition limits propagule supply, the second strongly filters seedling establishment, and both create impoverished forest understories composed of few individuals and species. In undisturbed forest understories, the inertia toward impoverishment was sustained across the decade even where browsing was eliminated. Consequently, stand replacing disturbances (whether natural or anthropogenic) that disrupt the understory layer and reinitiate succession may be necessary to propel forests out of their current stasis and down a pathway leading to greater diversity.