X-ray diffraction method was applied to measure residual stresses and stored elastic energy in deformed and annealed polycrystalline ferritic and austenitic steel samples. Orientation distribution of plastic incompatibility second order stresses created during elastoplastic deformation was determined and presented in Euler space. Using deformation models, these stresses were correlated with different types of intergranular interactions occurring in the studied materials. An important decrease of the first and the second order residual stresses was observed during recovery and recrystallization processes. Also diffraction peak widths, related to dislocations density, were studied and correlated with stress variation during annealing process. Differences in stress relaxation between ferritic and austenitic samples were explained by different values of the stacking fault energy, which influences dislocation climb and cross-slip.