Early sensory experience during the critical period of development is known to alter sensory cortical response properties and organization. We find that the earliest sound driven activity in the mouse auditory cortex (ACX) can be detected before earcanal opening (ECO). How auditory experience at these ages may influence auditory cortical development is unknown. Particularly, subplate neurons (SPNs), involved in sensory thalamocortical maturation, in the auditory cortex, are also found to be driven by sounds at ages before ECO in mice. We find that SPNs are selective to low probability deviant sounds in auditory streams before ECO, more so than thalamo-recipient, layer 4 (L4) neurons and not after ECO. We hypothesize that SPNs with their deviant selectivity can direct development of L4 responses before ECO. Exposing mice before ECO with a rarely occurring tone in a stream of another tone occurring frequently leads to strengthening of cortical representation of the rare tone, but not the frequent tone in the adult. Control exposure experiments with only a frequently occurring tone and also in other developmental age windows corroborate the importance of low probability sounds in auditory development. With a computational network model of known thalamic inputs to SPNs and L4 we explain the observed developmental plasticity. An information theoretic analysis with sparse coding assumptions also predicts the observations. Thus, salient low probability sounds in the earliest auditory environment causes long term changes in the auditory cortex.