The fitness of groups is often considered to be the average fitness among constituent members. This assumption has been useful for developing models of multilevel selection, but its uncritical adoption has held back our understanding of how multilevel selection actually works in nature. If group fitness is only equal to mean member fitness, then it is a simple task to erode the importance of group-level selection in all multilevel scenarios—species selection could then be reduced to organismal selection as easily as group selection can. Because selection from different levels can act on a single trait, body size, for example, there must be a way to translate one level of fitness to another. This allows the calculation of the contributions of selection at each level. If high-level fitness is not a simple function of member fitness, then how do they interlace? Here we reintroduce Leigh Van Valen’s argument for the inclusion of expansion as a component of fitness. We show that expansion is an integral part of fitness even if one does not subscribe to the energetic view of fitness from which Van Valen originally derived it. From a hierarchical perspective, expansion is the projection of demographic fitness from one level to the next level up; differential births and deaths at one level produce differential expansion one level above. Including expansion in our conceptual tool kit helps allay concerns about our ability to identify the level of selection using a number of methods as well as allowing for the various forms of multilevel selection to be seen as manifestations of the same basic process.