Permeability is the most important property that controls the transfer of gas mass across a hierarchy of scales within a shale gas reservoir. When gas diffuses from the fracture wall into the matrix, the gas adsorbs onto shale grains. This adsorption may result in matrix swelling. In previous studies, it is commonly assumed that this swelling is uniform within the matrix. Under this assumption, the impact of the gas diffusion process would be neglectable. In this study, we hypothesize that this uniform swelling assumption is responsible for the inconsistencies between poroelastic solutions and experimental or field observations as reported in the literature. We introduce a volumetric ratio of the gas-invaded volume to the whole matrix volume to quantify the impact of matrix swelling volume expansion on the evolution of shale permeability. The gradual matrix pressure increase in the vicinity of fracture walls leads to local swelling. As the gas invaded zone expands within the matrix, the local effect weakens. When the matrix is completely invaded by the injected gas, a new homogeneous state is achieved, and the local effect ends. We find that the evolution of shale permeability from initial to final homogeneous states is a result of the propagation of the gas invaded area. We apply this approach to generate a series of shale permeability maps. These maps explain experimental observations under a spectrum of conditions from constant confining pressure, to constant average pore pressure, to constant effective stress, and to constant total volume conditions.