Despite significant progress in understanding neural coding, it remains unclear how the coordinated activity of large populations of neurons relates to what an observer actually perceives. Since neurophysiological differences must underlie differences among percepts, differentiation analysis—quantifying distinct patterns of neurophysiological activity—is an “inside out” approach that addresses this question. We used two-photon calcium imaging in mice to systematically survey stimulus-evoked neurophysiological differentiation in excitatory populations across 3 cortical layers (L2/3, L4, and L5) in each of 5 visual cortical areas (primary, lateral, anterolateral, posteromedial, and anteromedial) in response to naturalistic and phase-scrambled movie stimuli. We find that unscrambled stimuli evoke greater neurophysiological differentiation than scrambled stimuli specifically in L2/3 of the anterolateral and anteromedial areas, and that this effect is modulated by arousal state and locomotion. Contrariwise, decoding performance was far above chance and did not vary substantially across areas and layers. Differentiation also differed within the unscrambled stimulus set, suggesting that differentiation analysis may be used to probe the ethological relevance of individual stimuli.