Individuals differ in their ability to selectively attend to goal-relevant auditory stimuli. People with Attention-Deficit/Hyperactivity Disorder (ADHD) in particular tend to show cognitive deficits associated with distractibility and inefficiencies in inhibition and attention. We hypothesized that people with ADHD would exhibit poorer performance and weaker neural signatures of attentional control when undertaking a challenging auditory task that required strong top-down attention. Neurotypical (N = 20) and ADHD (N = 25) young adults with normal hearing listened to one of three concurrent, spatially separated speech streams and reported the order of the syllables presented while we recorded electroencephalography (EEG). We tested both the ability to sustain attentional focus on a single "target" stream and the ability to monitor the target but flexibly switch attention to an unpredictable "interrupter" stream from another direction if and when it appeared. Although both stimulus structure and task demands affected behavioral performance, ADHD status did not. In both groups, the interrupter evoked larger neural responses when it was to be attended compared to when it was irrelevant, including for the P3a "reorienting" response previously described as involuntary. This attentional modulation was weaker in ADHD listeners, even though their behavioral performance was no lower. Across the entire cohort, individual performance correlated with the degree of top-down modulation of neural responses. These results demonstrate that listeners differ in their ability to modulate neural representations of sound based on task goals. Adults with ADHD have weaker volitional control of attentional processes than their neurotypical counterparts.