Understanding fluxes and instabilities within freestanding ultrathin films is necessary for a better understanding of, and control over, the stability and lifetime of foams and emulsions. In micellar foam films, confinement-induced layering of micelles leads to stepwise thinning or stratification that occurs by the expansion of thinner, darker domains. Often, because of a nanoridge-to-mesa instability, one or more brighter white spots or "mesas" appear at the circular moving front between thinner domains and the thicker (less dark) surrounding film. Previous studies assume that the instability and the appearance of white spots are similar to the capillarity-driven Rayleigh instability that leads to the breakup of a coherent liquid jet. Using the IDIOM (interferometry digital imaging optical microscopy) protocols we recently developed, we characterize the nanoridge-to-mesa instability with exquisite spatiotemporal resolution (thickness <1 nm, time <1 ms). The instability could be classified as a Rayleigh instability if a similar sequence of thick and thin undulations is visualized around the expanding domains. However, quantitative analysis reveals that only mesas grow in size after the instability, whereas the rest of the nanoridge preserves its shape. By analogy to the phase separation into compositionally distinct regions, we show that the spontaneous nucleation of thicker mesas in stratifying films is a phase transition driven by the oscillatory nature of the free-energy functional.