The interlocking and packing of aggregate particles play a key role in achieving high level of linear viscoelastic properties and rutting resistance in asphalt mix for sustainable pavements. In this study, the quantitative effect of fractured aggregate particles (FAPs) on loading (i.e., 500 kPa normal stress), along with the resilience properties of asphalt mixes, was evaluated. Linear and nonlinear stress behaviors of aggregates (from direct shear test) and asphalt mixes (from resilient modulus test) were analyzed. A new quantitative parameter (linear stress ratio), i.e., linear stress (Pi)/maximum stress (Pmax), is proposed to be used in selection of aggregates for asphalt mixes. It was observed that 15.5% increase in FAPs caused 19.5% increase in ϕ and 70.7% increase in linear stress ratio (LSR). The same content of FAPs resulted in 29.4% and 36% increases in total resilient modulus (MRT) and 34.2% and 24.5% increases in instantaneous resilient modulus (MRI) for 0.1 and 0.3 s load durations, respectively. The proposed LSR is observed to be superior to conventionally used ϕ for aggregate resistance in design of flexible pavements.