In this paper, nitrogen diffusion is investigated in single-crystalline austenitic stainless steel during modified layer formation and thermal annealing. A generalized system of diffusion equations is derived within a thermodynamic framework from Fick’s laws, which describe nitrogen flux under multiple driving forces, including a concentration gradient and the gradient of hydrostatic stress. Trapping and detrapping phenomena are considered within this model, and nitrogen flux is distinguished depending on whether nitrogen is in a lattice or a trapping site. Furthermore, the effects of anisotropic elasticity in single-crystal austenitic stainless steel on the stress field are investigated. The proposed model is used to simulate the nitrogen transportation process in single-crystalline AISI 316L during ion beam nitriding and after isothermal annealing at three different crystalline orientations. The results of our theoretical predictions are compared with experimental results taken from the literature. It is shown that during isothermal annealing, nitrogen diffusion becomes significantly slower than during nitriding. The diffusion coefficient during the annealing process, compared with the nitriding process, decreases by factors of 4.3, 3.3, and 2.5 for the orientations (001), (011), and (111), respectively.