Fine aggregate matrix (FAM), as the matrix phase in asphalt concrete (AC), significantly affects the fatigue behavior of AC. To accurately assess the mechanical properties of FAM, a newly designed experimental strategy for FAM testing was developed, and the viscoelastic continuum damage theory (VECD) theory was applied to analyze FAM’s fatigue cracking characteristics. In this study, a dumbbell-shaped geometry for dynamic shear rheometer testing was designed and verified through the FE-aided method. Subsequently, three AC mixtures’ FAM specimens with this special geometry were fabricated for the frequency sweep and linear amplitude sweep tests. Results showed that the specially designed specimens effectively capture the viscoelastic and fatigue properties of FAM with high replicability. Analyses based on the VECD theory indicated that FAM of porous asphalt (FAM(PA13)), featuring a higher asphalt content, exhibits a significant reduction in pseudo stiffness with increasing damage at the initial stage, but the reduction rate diminishes as damage progresses when compared to the other two FAMs. It was speculated that the higher aggregate content in FAM of dense-graded AC mixture (FAM(AC20) induces stress concentrations in the asphalt mastic near the protrusion areas of aggregates, thereby rendering the sample more susceptible to damage. The proposed methods will be readily extended to characterize other mechanical properties of FAM.