To investigate the origin of heterogeneous water distribution in the suboceanic lithospheric mantle, we performed a detailed petrological and geochemical analysis of abyssal peridotites collected from two localities (53°E and 63.5°E) on the Southwest Indian Ridge. These serpentinized peridotites display primary olivine‐orthopyroxene‐clinopyroxene‐spinel assemblage and record equilibrium temperature of 1150–1200°C and around 1000°C for the 53°E and 63.5°E locations, respectively. The rocks were thus equilibrated in the spinel stability field prior to their exhumation to the seafloor. Our FTIR analyses show variable water contents in orthopyroxene ranging from 24 to 262 wt ppm H2O. Orthopyroxene in the 63.5°E peridotites is characterized by homogeneous and high water content (>200 ppm), whereas orthopyroxene in the 53°E peridotites displays a wider range of water contents (24–246 ppm). We first demonstrate that differences in equilibrium conditions (i.e., pressure and temperature) and mineral chemistry and serpentinization cannot explain the water content variations. Melting modeling show that a fractional melting in both garnet and spinel stability fields is needed to explain the MREE and HREE concentrations in clinopyroxene. Enrichment in LREE, high water contents, and high H2O/Ce, however, require a postmelting rehydration event such as metasomatism. Based on petrographic evidence and investigation of chemical heterogeneities at segment/dredge scale, we suggest that this event involves a metasomatic agent enriched in water and incompatible elements. We infer that the small‐scale heterogeneities in water and trace elements may result either from successive infiltration of various amounts of melt/fluids as described in the formation of oceanic core complex or from spatial heterogeneities of melt infiltration as observed in peridotite massifs.