During the self-piercing riveting (SPR) process, residual stress develops due to the high plastic deformation of the sheet materials. In this study, the effect of the residual stress on the fatigue lifetime of SPR joints with dissimilar magnesium AZ31 alloy and aluminum Al5052 alloy sheets was evaluated. The residual stress distribution was derived through a simulation of the SPR process by the FEA (finite element analysis). The measured values by the X-ray diffraction technique confirmed that the validity of the simulation has a maximum error of 17.2% with the experimental results. The fatigue strength of the SPR joint was evaluated at various loading angles using tensile-shear and cross-shaped specimens. It was found that the compressive residual stresses of the joint reduce the stress amplitude by 13% at 106 cycles lifetime, resulting in extension of its lifetime to approximately 3.4 million cycles from 106 cycles lifetime. Finally, it was confirmed that the fatigue life of SPR joints was appropriately predicted within a factor of three using the relationship between the fatigue life and the equivalent stress intensity factor. The fatigue resistance of the magnesium AZ31 alloy on the upper sheet was found to govern fatigue lifetimes of SPR joints of dissimilar magnesium AZ31 alloy sheets.