The present work proposes a suitable approach for predicting ductile fracture in a new joining process by plastic deformation called hole hemming. This process creates a combined form- and force-fit joint and enables the joining of lightweight materials with varying formability without requiring heating or auxiliary elements. In this process, the joinability of materials is limited by the occurrence of fracture in the outer sheet, highlighting the crucial need to accurately predict its damage during the process design phase. In this study, five different fracture criteria, including the McClintock, Rice–Tracey, Normalized Cockroft–Latham, and Brozzo and Modified Mohr–Coulomb (MMC) criteria, are examined during the joining of a challenging combination of lightweight materials (aluminum AA6082-T4 and magnesium AZ31). These criteria are calibrated by a hybrid experimental–numerical method using three tests with distinct stress states. These criteria are then implemented into the finite element model of the hole hemming process, utilizing an appropriate user subroutine. The results show that the flange edge of the outer sheet is the most prone region to fracture during the joining process, and a criterion must be able to model the fracture behavior of the material from uniaxial tension to shear to accurately predict fracture in this area. Among the examined criteria, only the MMC criterion was capable of such modeling and accurately predicted the critical displacement of the punch in the hemming stage with a negligible error (about 1%). On the other hand, the prediction accuracy of the other criteria varied significantly depending on the calibration test, resulting in errors ranging from 8.6% to 75.5%. The error of 8.6% was achieved with the Normalized Cockroft–Latham criterion calibrated by a uniaxial tension test. Thus, based on the results, the MMC criterion is recommended for ductile fracture prediction in the hole hemming process, offering valuable insights to assist in process design.