The relationship between dilatancy and anisotropy is a fundamental aspect of anisotropic behavior of granular materials. Existing test data directly investigating this relationship are scarce and conflicting. Discrete element biaxial and triaxial numerical tests on idealized granular materials in both two-dimensional (2D) and three-dimensional (3D) are conducted in this study to acquire high quality stress, strain, dilatancy, and fabric data for various anisotropic samples, which are utilized to analyze the dependency of dilatancy ratio on fabric anisotropy. The test results indicate that the dilatancy ratio is not only dependent on the initial fabric anisotropy, but is also influenced by the evolution of fabric, especially the contact normal fabric. At low deviatoric stress ratio under biaxial and triaxial loading, difference in initial fabric anisotropy of granular materials can lead to distinctly different dilatancy ratios. As loading continues, the deviatoric stress ratio, void ratio, and fabric of granular materials evolve toward the unique critical state, causing the dilatancy ratio to converge irrespective of its initial value. The anisotropic critical state theory (ACST) is shown to be capable of providing a framework for quantitative mathematical depiction of the dependency of dilatancy ratio on fabric anisotropy.