The effect of some important testing conditions including test specimen (SCB and ENDB), fracture mode (I and II), loading rate (0.5, 1.0, and 5.0 mm/min), and temperature (−5°C, −15°C, and −25°C) is evaluated on fracture toughness of asphalt material containing different percentages of fibers (0%, 0.025%, 0.05%, and 0.1% of unit weight). The experimental and numerical results show that the addition of 0.025%, 0.05%, and 0.1% of the polyolefin–aramid fiber in asphalt mixture increases the fracture toughness of the control mixture by about 8%, 17%, and 23% in pure mode I and 7%, 16%, and 24% in pure mode II, respectively; which are among the most promising enhancements compared to other fibers. Results also reveal that fracture toughness at mode I obtained from the SCB and ENDB specimens is similar. However, the mode II fracture toughness obtained from the ENDB specimen is about 12% higher than the corresponding value obtained from the SCB specimen.