The formation of White (WEL) and Brown Etching Layers (BEL) on rail raceways during service causes the initiation of microcracks which finally leads to failure. Detailed characterization of the WEL and the BEL in a pearlitic rail steel is carried out from micrometer to atomic scale to understand their microstructural evolution. A microstructural gradient is observed along the rail depth including martensite, austenite and partially dissolved parent cementite in the WEL and tempered martensite, ultrafine/nanocrystalline martensite/austenite, carbon saturated ferrite and partially dissolved parent cementite in the BEL. Plastic deformation in combination with a temperature rise during wheel-rail contact was found to be responsible for the initial formation and further microstructural evolution of these layers. The presence of austenite in the WEL/BEL proves experimentally that temperatures rise into the austenite range during wheel-rail contact. This is in agreement with finite element modelling results. Each wheel-rail contact must be considered as an individual short but intense deformation and heat treatment cycle that cumulatively forms the final microstructure, as shown by diffusion length calculations of C and Mn. The presence of secondary carbides in the BEL indicates that the temperature in the BEL during individual loading cycles reaches levels where martensite tempering occurs. Partially fragmented primary cementite laths, enriched in Mn, depleted in Si, and surrounded by a C-gradient and dislocations were found in the BEL. The initial step in the formation of BEL and WEL is the defect-and diffusion-assisted decomposition of the original microstructure.