Permeability damage in subsurface porous media caused by clay mobilization is encountered in many engineering applications, such as geothermal energy, water disposal, oil recovery, and underground CO2 storage. During the freshwater injection into rocks containing brine, the sudden decrease in salinity causes native clay fines to detach and clog pore throats, leading to a significant decline in permeability. The clay fines detach due to weakened net-attractive forces binding them to each other and the grain. Past experiments link this permeability damage on the immediate history of the salinity and the direction of flow. To better understand this phenomenon, we conducted pore-scale simulations of cyclic injection of freshwater and brine into sandstone containing Kaolinite clay. Our simulations establish a link between the clay-fine trajectory and the permeability trend observed by Khilar and Fogler (1983). For a uniform clay size of 3 microns, we observe a permeability decline by two orders of magnitude during freshwater injection with respect to brine injection. Increasing salinity and simultaneously reversing flow direction restores the permeability. The permeability restoration upon reversing the brine flow direction is attributed to the unblocking of pore throats in the reverse direction by the movement of the clay particles along the grain surfaces by the hydrodynamic force and the strong net-attractive force under high salinity.