The phenotypic differences between the sexes are generated by genes with sex-biased expression. These range from a few major regulators to large numbers of organ-specific effector genes in sexually mature individuals. We explore the variation and evolutionary patterns of these genes in a large dataset from natural populations of sub-species and species of mice across an evolutionary distance of 2 million years. Intriguingly, even within these short phylogenetic distances, we find an extremely fast evolutionary turnover of sex-biased gene expression and fast adaptive protein evolution. To capture the individual variances of sex-biased expression, we have developed a sex-biased gene expression index (SBI) that represents the cumulative expression of all sex-biased genes for each individual in each organ. We find that SBI distributions are often overlapping between the sexes and do not correlate between organs, thus defying a simple binary sex distinction for given individuals. Comparison with data from humans shows fewer sex-biased genes in most organs and strongly overlapping SBI distributions between the sexes. We conclude that sex-biased genes are subject to particularly fast evolution, with no long-term stability for male or female expression characteristics and that individual variation in sex-related phenotypic characters is usually overlapping between the sexes.