The objective of this study is to investigate automatic recognition of speech-evoked auditory brainstem responses (speech-evoked ABR) to the five English vowels (/a/, /ae/, /ao (ɔ)/, /i/ and /u/). We used different automatic speech recognition methods to discriminate between the responses to the vowels. The best recognition result was obtained by applying principal component analysis (PCA) on the amplitudes of the first ten harmonic components of the envelope following response (based on spectral components at fundamental frequency and its harmonics) and of the frequency following response (based on spectral components in first formant region) and combining these two feature sets. With this combined feature set used as input to an artificial neural network, a recognition accuracy of 83.8% was achieved. This study could be extended to more complex stimuli to improve assessment of the auditory system for speech communication in hearing impaired individuals, and potentially help in the objective fitting of hearing aids.iii