Asphalt mixtures are widely used as a surfacing material for pavements due to their several advantages. For this reason, robust numerical models still need to be developed to improve the understanding of their fracture behaviour. Recently, an incremental generalised Kelvin (GK) contact model that relates increments in contact displacements with increments in contact forces was proposed to assess the viscoelastic behaviour of asphalt mixtures within a discrete element method (DEM) framework. In this work, the contact model is extended to allow its application to asphalt mixture fracture studies. Two damage models—a brittle and a bilinear softening—coupled with the GK contact model are proposed to consider damage initiation and propagation. A parametric study is presented that assesses the impact of the GK-Damage parameters, showing a sensitivity to the loading velocity and the Maxwell elements, particularly its viscosity element, on the stress–strain response of a single contact. A reduced-size numerical mastic is initially used to speed up the calibration process of the GK-Damage contact parameters, with subsequent validation on a specimen with real experimental dimensions. It is shown that the proposed calibrated damage models can successfully reproduce the time-dependent behaviour, peak stress, and crack path observed in experimental results, highlighting the benefits of the adopted methodology. For the GK-Bilinear model, the fracture energy and maximum contact tensile stress are shown to adjust both the peak stress and softening response. Uniaxial tensile tests on asphalt mixtures indicate that the GK-Bilinear model provides a more realistic characterisation of fracture development. A higher susceptibility to damage at aggregate-to-mastic contacts compared to contacts within the mastic phase is identified.