The microstructural evolution during refill friction stir spot welding of an AlMgSc alloy was studied. The primary texture that developed in all regions, with the exception of the weld center, was determined to be h110i fibers and interpreted as a simple shear texture with the h110i direction aligned with the shear direction. The material flow is mainly driven by two components: the simple shear acting on the horizontal plane causing an inward-directed spiral flow and the extrusion acting on the vertical plane causing an upward-directed or downward-directed flow. Under such a complex material flow, the weld center, which is subjected to minimal local strain, is the least recrystallized. In addition to the geometric effects of strain and grain subdivision, thermally activated high-angle grain boundary migration, particularly continuous dynamic recrystallization, drives the formation of refined grains in the stirred zone.