Previous studies have suggested that varying attention demands in dichotic listening (DL) tasks might be a clinically feasible method to distinguish ‘bottom-up’ from ‘top-down’ deficits in listening. This study aims to investigate DL processing in adults with listening difficulties (LD). We assessed the performance of a listening difficulties group (LDG) (n = 24, mean age = 24, backward digit span = 4.0) and a control group (CG) (n = 25, mean age = 29.2, backward digit span = 6.4) in DL tests involving non-forced and both right and left-forced attention. The results indicated an overall significantly worse performance of LDG compared to the CG, which was greater for forced-left condition. This same result was observed when controlling for working memory (WM) variance. Both groups presented an overall right ear advantage with no difference in terms of the magnitude of advantage. These results indicate that LD presented by the LDG might be due to a combination of sensory and cognitive deficits, with emphasis on the cognitive component. However, the WM, although impaired in the LDG group, was not the main factor in segregating both groups. The role of the additional cognitive processes such as inhibitory control in LD is discussed.