Prematurity is one of the most crucial risk factors negatively affecting the maturation of the auditory system. Children born preterm demonstrate high rates of hearing impairments. Auditory processing difficulties in preterm children might be a result of disturbances in the central auditory system development and/or sensory deprivation due to peripheral hearing loss. To investigate auditory processing in preterm children, we utilized a set of psychoacoustic tests to assess temporal processing and speech intelligibility. A total of 241 children aged 6–11 years old (136 born preterm and 105 healthy full-term children forming the control group) were assessed. The preterm children were divided into three groups based on their peripheral hearing status: 74 normal hearing (NH group); 30 children with bilateral permanent sensorineural hearing loss (SNHL group) and 32 children with bilateral auditory neuropathy spectrum disorder (ANSD group). The results showed significantly worse performance in all tests in premature children compared with full-term children. NH and SNHL groups showed significant age-related improvement in speech recognition thresholds in noise that might signify a “bottom-up” auditory processing maturation effect. Overall, all premature children had signs of auditory processing disorders of varying degrees. Analyzing and understanding the auditory processing specificity in preterm children can positively contribute to the more effective implementation of rehabilitation programs.