Background: Idiopathic hypogonadotropic hypogonadism (IHH) is rare and can either be associated with normal or defective olfactory sensation, classified as normosmic IHH (nIHH) or Kallmann’s syndrome (KS), respectively. We do not yet understand the central processing pathways in the olfactory system, especially regarding these disorders. We aimed to compare the resting-state structural and functional connectivity (FC) of olfactory neural pathways in patients with nIHH and KS.Methods: A total of 50 males were studied: 13 nIHH patients, 12 KS patients, and 25 healthy controls (HCs). All subjects underwent diffusion tensor imaging (DTI) and functional magnetic resonance imaging (fMRI) scans. Structural and functional connectivity data analyses were then performed.Results: The results indicated that fractional anisotropy (FA) was significantly decreased in the right uncinate fasciculus (UF) in the KS group. The olfactory cortex FC values of the right gyrus rectus and orbitofrontal cortex (OFC) in the KS group were decreased compared with those in the HC group and increased compared with those in the nIHH group (nIHH< KS <HC). Moreover, there were significant negative correlations between right UF FA and olfactory cortex FC to both the gyrus rectus and OFC within the nIHH and HC groups.Conclusion: We have reported significant structural and functional disruptions unilaterally at the right junction of the fronto-limbic system in KS patients. The results may indicate that a specific structural-functional asymmetry exists in the olfactory cortex pathways in KS patients.