Evolutionary biologists since Darwin have hypothesized that closely related species compete more intensely and are therefore less likely to coexist. However, recent theory posits that species diverge in two ways: either through the evolution of 'stabilizing differences' that promote coexistence by causing individuals to compete more strongly with conspecifics than individuals of other species, or through the evolution of 'fitness differences' that cause species to differ in competitive ability and lead to exclusion of the weaker competitor. We tested macroevolutionary patterns of divergence by competing pairs of annual plant species that differ in their phylogenetic relationships, and in whether they have historically occurred in the same region or different regions (sympatric versus allopatric occurrence). For sympatrically occurring species pairs, stabilizing differences rapidly increased with phylogenetic distance. However, fitness differences also increased with phylogenetic distance, resulting in coexistence outcomes that were unpredictable based on phylogenetic relationships. For allopatric species, stabilizing differences showed no trend with phylogenetic distance, whereas fitness differences increased, causing coexistence to become less likely among distant relatives. Our results illustrate the role of species' historical interactions in shaping how phylogenetic relationships structure competitive dynamics, and offer an explanation for the evolution of invasion potential of non-native species.