In the context of neurophysiological normative data, it has been established that aging has a significant impact on neurophysiological correlates of auditory phonological input processes, such as phoneme discrimination (PD) and word recognition (WR). Besides age, sex is another demographic factor that influences several language processes. We aimed to disentangle whether sex has a similar effect on PD and WR. Event-related potentials (ERPs) were recorded in 20 men and 24 women. During PD, three phonemic contrasts (place and manner of articulation and voicing) were compared using the attentive P300 and pre-attentive Mismatch Negativity. To investigate WR, real words were contrasted with pseudowords in a pre-attentive oddball task. Women demonstrated a larger sensitivity to spectrotemporal differences, as evidenced by larger P300 responses to the place of articulation (PoA) contrast and larger P300 and MMN responses than men in PoA-based PD. Men did not display such sensitivity. Attention played an important role, considering that women needed more attentional resources to differentiate between PoA and the other phonemic contrasts. During WR, pseudowords evoked larger amplitudes already 100 ms post-stimulus independent of sex. However, women had decreased P200 latencies, but longer N400 latencies in response to pseudowords, whereas men showed increased N400 latencies compared to women in response to real words. The current results demonstrate significant sex-related influences on phonological input processes. Therefore, existing neurophysiological normative data for age should be complemented for the factor sex.