With a view to improving the surface quality of the magnesium (Mg) alloy at a warm temperature, the ultrasonic-assisted warm single point incremental forming (SPIF) is proposed. The surface quality of the Mg alloy during the warm SPIF is primarily affected by two parts: the orange peel patterns of the non-contact surface and the scratches and adhesive wear of the contact surface. In this work, the surface quality of the AZ31B Mg alloy sheet parts at different forming temperatures and ultrasonic amplitudes is evaluated by examining the surface roughness and topography. The results show that the generation of orange peel patterns is significantly affected by temperature. In addition, scratches and adhesive wear of the contact surface increase with the rising temperature. After applying the ultrasonic vibration (UV), the quality of both the non-contact and the contact surfaces of the parts are significantly improved, but too large ultrasonic amplitude slightly reduces the surface quality. Moreover, the microstructural examination results show that UV has a great effect on dynamic recrystallization and grain refinement, which positively affects surface quality improvement.