Cortical remapping after hand loss in the primary somatosensory cortex (S1) is thought to be predominantly dictated by cortical proximity, with adjacent body parts remapping into the deprived area. Traditionally, this remapping has been characterised by changes in the lip representation, which is assumed to be the immediate neighbour of the hand based on electrophysiological research in non-human primates. However, the orientation of facial somatotopy in humans is debated, with contrasting work reporting both an inverted and upright topography. We aimed to fill this gap in the S1 homunculus by investigating the topographic organisation of the face. Using both univariate and multivariate approaches we examined the extent of face-to-hand remapping in individuals with a congenital and acquired missing hand (hereafter one-handers and amputees, respectively), relative to two-handed controls. Participants were asked to move different facial parts (forehead, nose, lips, tongue) during fMRI scanning. We first report evidence for an upright facial organisation in all three groups, with the upper face and not the lips bordering the hand area. We further found little evidence for remapping of all tested facial parts in amputees, with no significant relationship to the chronicity of their PLP. In contrast, we found converging evidence for a complex pattern of face remapping in congenital one-handers across all facial parts, where the location of the cortical neighbour, the forehead, is shown to shift away from the deprived hand area, which is subsequently activated by the lips and the tongue. Together, our findings demonstrate that the face representation in humans is highly plastic, but that this plasticity is restricted by the developmental stage of input deprivation, rather than cortical proximity.