This paper presents the results of a research study and analysis conducted to determine the degree of anisotropy of asphalt concrete in terms of its initial elastic properties. The analysis of asphalt concrete was focused on determining the effective constrained stiffness modulus in three mutually perpendicular directions based on the finite element method. The internal structure of the asphalt concrete was divided into the mortar phase and the mineral aggregate phase. Static creep tests using the Bending Beam Rheometer were conducted for the mortar phase to fit the rheological model. The aggregate arrangement and orientation were analysed using an image analytical technique for the mineral phase. The Finite Element Method (FEM) meshes were prepared based on grey images with an assumption of plane strain in 2D formulation. Using the FEM model, the tension/compression tests using selected characteristic directions were conducted, and the effective constrained stiffness moduli were estimated. This study showed a dominant horizontal direction for all coarse aggregates resulting from the normal force of the road roller and paving machines during laying and compaction on a road site. Depending on the values of the mortar’s mechanical parameters and the load direction, the effective stiffness modulus might differ by ±20%. Based on the FEM analysis, this result was proven and commented on through an effective directional modulus evaluation and a presentation of internal stress distribution. Depending on the shape and orientation of the aggregates, it was possible to observe local “stress bridging” (transferring stresses from aggregate to aggregate when contacting). Moreover, the rheological properties of the mortar were considered by assuming two limiting situations (instantaneous and relaxed moduli), determining the bands of all possible solutions. In the performed FEM analysis, the influence of the Poisson ratio was also considered. The analysed asphalt concrete tends to be isotropic when the Poisson’s mortar ratio is close to the value of 0.5, which agrees with the physical expectations. The obtained results are limited to particular asphalt concrete and should not be extrapolated to other asphalt mixture types without prior analysis.