The plate-to-sleeve welded joint is employed in the aluminum alloy spatial structure. The welding deformation and residual stress determine the dimensional accuracy and even increase the failure risk. Based on ABAQUS software and the coordinate transformation method, a finite element model with the arc weaving was developed to investigate the temperature field, welding deformation, and residual stress of an AA6061-T6 plate-to-sleeve welded joint with 8-pass. The metal inert gas welding experiment was performed with the corresponding joint, and the welding deformation and residual stresses were measured to verify the numerical results. The results show that the thermal cycle of the weaving weld and its vicinity have multiple peaks, and the time interval between two adjacent peaks is one weaving period. The deformation pattern of the joint is angular deformation. The maximum deformation is observed in the weld toe of the top surface. The longitudinal residual stress in the weld and its vicinity is high tensile stress, and that in the upper part of the sleeve is mainly compressive stress. The transverse residual stress in the weld and its vicinity of the top surface is low tensile stress, and that in the bottom surface is compressive stress. Furthermore, the evolution of the welding deformation and residual stresses attributed to subsequent weld passes was elaborated, which provides a reference for developing control schemes.