Diagnosis of the dynamic response of an asphalt pavement structure coupled with involving inherent anisotropic properties of pavement materials serves as a vital tool for pavement analysis and design platforms. The mechanical response of asphalt pavement is strongly influenced by the random anisotropic properties (simplified as transversely isotropic properties in this study) because of the shape, distribution, orientation, degree of compaction, void structure, and so forth, of the granular materials in asphalt pavements. This study aims to introduce a computational framework by generating a three-dimensional finite element-based program to analyze the influence of thermo-viscoelasticity of the asphalt layer and transverse isotropy of the base courses and its randomness on asphalt pavements under the effect of moving vehicle loading. The accuracy and efficiency of the developed numerical program were verified by comparing our results with previous studies. Concurrently, the influences of random field conditions, transversely isotropic properties, and the temperature field were involved in assessing their action on the fatigue life prediction of pavement. It was concluded that the fatigue life of the asphalt pavement model, considering the transverse isotropy of the material, random field, and temperature field, was reduced by 48.1%, which would mislead the state assessment of asphalt pavements. Therefore, during asphalt pavement design and its viscoelastic response analysis, it is recommended to consider the influence of the random modulus, temperature fields, and transversely isotropic properties on the structural assessments.