Individuals with congenital sensory loss experience a total lack of input from the lost sense throughout development. This lack of input has been linked to altered brain morphology, mainly in cerebral areas normally devoted to the processing of the lost sense. In congenital olfactory sensory loss (anosmia), the only consistent morphological finding is a decreased depth of the olfactory sulci, linked to small or absent olfactory bulbs. Here, we aimed to establish whether congenital anosmia leads to more widespread alterations by comparing gray matter volume, cortical thickness, and curvature between 34 individuals with isolated congenital anosmia and 34 matched controls. Individuals with anosmia demonstrated alterations in bilateral olfactory sulci, encompassing decreased sulcus depth, gray matter atrophy, and decreased curvature. They further demonstrated increased gray matter volume and cortical thickness in the medial orbital gyri; regions strongly linked to olfactory processing, sensory integration and value coding. No structural alterations in the primary olfactory (piriform) cortex were, however, demonstrated. Our results replicate and extend previous knowledge with findings of divergent morphological alterations in bilateral orbitofrontal cortex, indicating influences of different plastic processes and suggesting that a lifelong absence of sensory input does not necessarily lead to morphological alterations in primary sensory cortex.