To date, numerous investigations have shown the beneficial effect of ultrasonic vibration-assisted forming technology due to its influence on the forming load, flow stress, friction condition reduction and the increase of the metal forming limit. Although the immediate occurring force and mean stress reduction are known phenomena, the underlying effects of ultrasonic-based material softening remain an object of current research. Therefore, in this article, we investigate the effect of upsetting with and without the ultrasonic vibrations (USV) on the evolution of the microstructure, stress relaxation and hardness of the AlMg3 aluminum alloy. To understand the process physics, after the UAC (ultrasonic assisted compression), the microstructures of the samples were analyzed by light and electron microscopy, including the orientation imaging via electron backscatter diffraction. According to the test result, it is found that ultrasonic vibration can reduce flow stress during the ultrasonic-assisted compression (UAC) process for the investigated aluminum–magnesium alloy due to the acoustic softening effect. By comparing the microstructures of samples compressed with and without simultaneous application of ultrasonic vibrations, the enhanced shear banding and grain rotation were found to be responsible for grain refinement enhancement. The coupled action of the ultrasonic vibrations and plastic deformation decreased the grains of AlMg3 alloy from ~270 μm to ~1.52 μm, which has resulted in a hardness enhancement of UAC processed sample to about 117 HV.