This study is aimed at examining the mesoscopic mechanical response and crack development characteristics of asphalt mixtures using the three-dimensional discrete element approach via particle flow code (PFC). The material is considered an assembly of three phases of aggregate, mortar, and voids, for which three types of contact are identified and described using a modified Burgers model allowing for bond failure and crack formation at contact. The laboratory splitting test is conducted to determine the contact parameters and to provide the basis for selecting three different load levels used in the indirect tensile fatigue test and simulation. The reliability of the simulation is verified by comparing the fatigue lives and dissipated energies against those from the test. Under cyclic loading, the internal tensile and compressive force chains vary dynamically as a response to the cyclic loading; both are initially concentrated beneath the top loading strip and then extend downward along the loading line. The compressive chains are oriented roughly vertically and develop an elliptic shape as damage grows, while the tensile chains are mostly horizontal and become denser. An analysis based on the histories of the numbers of different contact types indicates that damage mainly originates from bond failures among the aggregate particles and at the aggregate–mortar interfaces. In terms of location, cracking is initiated below the loading point (consistent with observations from the force chains) and propagates downward and laterally, leading to the macrocrack along the vertical diameter. The findings provide a mesoscopic understanding of the fatigue damage initiation and propagation in asphalt mixture.