Asphalt mixture is a type of textured composite material made of aggregates and mastic part. Overall strength and failure behavior in such materials depends on the texture or heterogeneity of the mixture. In particular, the crack growth mechanism from the tip of the pre-crack is significantly affected by the texture of the asphalt composite and environmental conditions. The crack can extend through the soft mastic, tight aggregates or interface of the mastic/aggregates. In this research, by performing some fracture tests on a typical asphalt mixture with different test specimens under mode I, mixed mode I/II and mixed mode I/III, the fracture resistance and trajectory of propagating crack is studied at two low and medium temperatures (i.e., −15 and +15 °C). The load bearing capacity and the fracture resistance of the tested asphalt samples increases by decreasing the temperature. It is also shown that a significant part of fracture plane passes through the soft mastic and boundary of aggregates (i.e., the interface of aggregates and mastic) and only about 10–15% of the fracture surface of the propagating crack passes via the tight aggregates by breaking them. This percentage decreases for mode II and III loading conditions and higher testing temperatures. Compared to brittle and isotropic materials, the fracture path of the asphalt mixture shows more deviation, and this deviation increases for those mixtures containing coarser aggregates in the ligament and tested under medium temperature conditions.