In this work, a 3D finite-element model (FEM) was developed to accurately predict the mechanical behavior of CFRP/Al self-piercing riveting (SPR) during the forming and shearing process. A continuous damage model based on the modified Hashin failure criterion was proposed to model the CFRP, and a cohesive model considering the traction-separation criterion was used to capture the delamination between CFRP layers. The simulation results were in good agreement with the experimental results. Based on the developed FEM, the damage of CFRP in the SPR forming process, the shear mechanical properties and damage of joints in the SPR shearing process, the effects of CFRP plate thickness and rivet length on the SPR forming cross-section and shear strength were investigated. The results showed that, the maximum interlock value of 0.46 mm was obtained when the rivet length was 6.5 mm and the CFRP plate thickness was 1.6 mm. The maximum shear peak load of 3611.98 N was obtained when the rivet length was 6.5 mm and the CFRP plate thickness was 2.2 mm. The interaction between rivet length and CFRP plate thickness should be considered in the actual SPR connection.