Fitness conferred by the same allele may differ between genotypes, and these differences shape variation and evolution. Changes in amino acid propensities at protein sites over the course of evolution have been inferred from sequence alignments statistically, but the existing methods are data-intensive and aggregate multiple sites. Here, we develop an approach to detect individual amino acids that confer different fitness in different groups of species from combined sequence and phylogenetic data. Our method is based on the mutual distribution of homoplasic (convergent or parallel) and divergent substitutions over a phylogeny, and uses the fact that a higher rate of homoplasic evolution within a clade signifies high fitness of the derived allele. We apply it to the env gene of two HIV-1 subtypes, A and B, and to the HA gene of two influenza A subtypes, H1 and H3, and show that the inferred fitness changes are consistent with the fitness differences observed in deep mutational scanning experiments. Changes in fitness do not always trigger episodes of positive selection, and therefore cannot be identified by existing methods for its detection.