Xylella fastidiosa infects several economically important crops in the Americas, and it also recently emerged in Europe. It is not clear, however, whether X. fastidiosa isolates display specialization to plant hosts and, if so, what genetic determinants underlie the associations. Here, using a set of Xylella genomes reflective of the genus-wide diversity, we performed a pan-genome analysis based on both core and accessory genes. We first investigated evolutionary relationships among isolates, recapitulating the three recognized X. fastidiosa subspecies based on information from both core and accessory genes. We used these phylogenies as the basis to test associations between genetic divergence and host species, yielding significant evidence for specialization of X. fastidiosa to plant host species for core, but not accessory, genes. This observation places constraint on the hypothesis that host specificity is conferred by shifts in accessory gene content. We also analyzed the ratio of nonsynonymous to synonymous substitutions in both core and accessory genes to identify positively selected genes that are potentially involved in arms-race dynamics. Arms-race genes may contribute to interactions with the plant hosts or insect vectors, and they are thus candidates to act as virulence or pathogenicity factors. We uncovered evidence of positive selection across codons of 19.5% (200 of 1,205) core genes and 7.7% (181 of 2,349) accessory genes. Most of these genes had unknown functions, but we identified some tractable candidates including nagZ_2, which encodes a beta-glucosidase that is important for biofilm formation Neisseria gonorrhoeae, and cya, which modulates gene expression in pathogenic bacteria.