BackgroundA next generation tomosynthesis (NGT) system, capable of two‐dimensional source motion, detector motion in the perpendicular direction, and magnification tomosynthesis, was constructed to investigate different acquisition geometries. Existing position‐based geometric calibration methods proved ineffective when applied to the NGT geometries.PurposeA line‐based iterative calibration method is developed to perform accurate geometric calibration for the NGT system.MethodsThe proposed method calculates the system geometry through virtual line segments created by pairs of fiducials within a calibration phantom, by minimizing the error between the line equations computed from the true and estimated fiducial projection pairs. It further attempts to correct the 3D fiducial locations based on the initial geometric calibration. The method's performance was assessed via simulation and experimental setups with four distinct NGT geometries: X, T, XZ, and TZ. The X geometry resembles a conventional DBT acquisition along the chest wall. The T geometry forms a “T”‐shaped source path in mediolateral (ML) and posteroanterior (PA) directions. A descending detector motion is added to both X and T geometries to form the XZ and TZ geometries, respectively. Simulation studies were conducted to assess the robustness of the method to geometric perturbations and inaccuracies in fiducial locations. Experimental studies were performed to assess the impact of phantom magnification and the performance of the proposed method for various geometries, compared to the traditional position‐based method. Star patterns were evaluated for both qualitative and quantitative analyses; the Fourier spectral distortions (FSDs) graphs and the contrast transfer function (CTF) were extracted. The limit of spatial resolution (LSR) was measured at 5% modulation of the CTF.ResultsThe proposed method presented is highly robust to geometric perturbation and fiducial inaccuracies. After the line‐based iterative method, the mean distance between the true and estimated fiducial projections was [X, T, XZ, TZ]: [0.01, 0.01, 0.02, 0.01] mm. The impact of phantom magnification was observed; a contact‐mode acquisition of a calibration phantom successfully provided an accurate geometry for 1.85× magnification images of a star pattern, with the X geometry. The FSD graphs for the contact‐mode T geometry acquisition presented evidence of super‐resolution, with the LSR of [0°‐quadrant: 8.57, 90°‐quadrant: 8.47] lp/mm. Finally, a contact‐mode XZ geometry acquisition and a 1.50× magnification TZ geometry acquisition were reconstructed with three calibration methods—position‐based, line‐based, and iterative line‐based. As more advanced methods are applied, the CTF becomes more isotropic, the FSD graphs demonstrate less spectral leakage as super‐resolution is achieved, and the degree of blurring artifacts reduces significantly.ConclusionsThis study introduces a robust calibration method tailored to the unique requirements of advanced tomosynthesis systems. By employing virtual line segments and iterative techniques, we ensure accurate geometric calibration while mitigating the limitations posed by the complex acquisition geometries of the NGT system. Our method's ability to handle various NGT configurations and its tolerance to fiducial misalignment make it a superior choice compared to traditional calibration techniques.