The outcomes of species interactions can vary greatly in time and space with the outcomes of some interactions determined by priority effects. On coral reefs, benthic algae rapidly colonize disturbed substrate. In the absence of top‐down control from herbivorous fishes, these algae can inhibit the recruitment of reef‐building corals, leading to a persistent phase shift to a macroalgae‐dominated state. Yet, corals may also inhibit colonization by macroalgae, and therefore the effects of herbivores on algal communities may be strongest following disturbances that reduce coral cover. Here, we report the results from experiments conducted on the fore reef of Moorea, French Polynesia, where we: (1) tested the ability of macroalgae to invade coral‐dominated and coral‐depauperate communities under different levels of herbivory, (2) explored the ability of juvenile corals (Pocillopora spp.) to suppress macroalgae, and (3) quantified the direct and indirect effects of fish herbivores and corallivores on juvenile corals. We found that macroalgae proliferated when herbivory was low but only in recently disturbed communities where coral cover was also low. When coral cover was <10%, macroalgae increased 20‐fold within 1 year under reduced herbivory conditions relative to high herbivory controls. Yet, when coral cover was high (50%), macroalgae were suppressed irrespective of the level of herbivory despite ample space for algal colonization. Once established in communities with low herbivory and low coral cover, macroalgae suppressed recruitment of coral larvae, reducing the capacity for coral replenishment. However, when we experimentally established small juvenile corals (2 cm diameter) following a disturbance, juvenile corals inhibited macroalgae from invading local neighborhoods, even in the absence of herbivores, indicating a strong priority effect in macroalgae–coral interactions. Surprisingly, fishes that initially facilitated coral recruitment by controlling algae had a net negative effect on juvenile corals via predation. Corallivores reduced the growth rates of corals exposed to fishes by ~30% relative to fish exclosures, despite increased competition with macroalgae within the exclosures. These results highlight that different processes are important for structuring coral reef ecosystems at different successional stages and underscore the need to consider multiple ecological processes and historical contingencies to predict coral community dynamics.