It remains poorly understood how brain causal connectivity networks change following hearing loss and their effects on cognition. In the current study, we investigated this issue. Twelve patients with long-term bilateral sensorineural hearing loss [mean age, 55.7 ± 2.0; range, 39–63 years; threshold of hearing level (HL): left ear, 49.0 ± 4.1 dB HL, range, 31.25–76.25 dB HL; right ear, 55.1 ± 7.1 dB HL, range, 35–115 dB HL; the duration of hearing loss, 16.67 ± 4.5, range, 3–55 years] and 12 matched normally hearing controls (mean age, 52.3 ± 1.8; range, 42–63 years; threshold of hearing level: left ear, 17.6 ± 1.3 dB HL, range, 11.25–26.25 dB HL; right ear, 19.7 ± 1.3 dB HL, range, 8.75–26.25 dB HL) participated in this experiment. We constructed and analyzed the causal connectivity networks based on functional magnetic resonance imaging data of these participants. Two-sample t-tests revealed significant changes of causal connections and nodal degrees in the right secondary visual cortex, associative visual cortex, right dorsolateral prefrontal cortex, left subgenual cortex, and the left cingulate cortex, as well as the shortest causal connectivity paths from the right secondary visual cortex to Broca’s area in hearing loss patients. Neuropsychological tests indicated that hearing loss patients presented significant cognitive decline. Pearson’s correlation analysis indicated that changes of nodal degrees and the shortest causal connectivity paths were significantly related with poor cognitive performances. We also found a cross-modal reorganization between associative visual cortex and auditory cortex in patients with hearing loss. Additionally, we noted that visual and auditory signals had different effects on neural activities of Broca’s area, respectively. These results suggest that changes in brain causal connectivity network are an important neuroimaging mark of cognitive decline. Our findings provide some implications for rehabilitation of hearing loss patients.