Generation and persistence of tinnitus following hearing loss may be due to aberrant engagement of attention. Here, functional MRI was used to determine differences in auditory and visual attention processing in adults with tinnitus and hearing loss compared to two age-matched control groups, one with matched hearing loss and the other with normal hearing thresholds. Attentional processing was investigated using a short-term memory task with varying loads, employing unfamiliar Korean letters in the visual condition and non-speech sounds in the auditory condition. We found similar behavioral response across the three groups for both modalities and tasks. For the auditory modality, the response of the attention network was suppressed in the tinnitus group compared to the control groups for both task loads, with the effect being more pronounced at high load. In contrast, in the visual modality, the tinnitus group exhibited greater response of the attention network, regardless of memory load, compared to the control groups. The results increase our understanding of the neural mechanisms of tinnitus and suggest that interventions that manipulate attention, especially in the visual domain, should be further investigated.