All lunar swirls are known to be co-located with crustal magnetic anomalies (LMAs). Not all LMAs can be associated with albedo markings, making swirls, and their possible connection with the former, an intriguing puzzle yet to be solved. By coupling fully kinetic simulations with a Surface Vector Mapping model, we show that solar wind standoff, an ion-electron kinetic interaction mechanism that locally prevents weathering by solar wind ions, reproduces the shape of the Reiner Gamma albedo pattern. Our method reveals why not every magnetic anomaly forms a distinct albedo marking. A qualitative match between optical remote observations and in situ particle measurements of the back-scattered ions is simultaneously achieved, demonstrating the importance of a kinetic approach to describe the solar wind interaction with LMAs. The anti-correlation between the predicted amount of surface weathering and the surface reflectance is strongest when evaluating the proton energy flux.