Van der Waals dielectrics are fundamental materials for condensed matter physics and advanced electronic applications. Most dielectrics host isotropic structures in crystalline or amorphous forms, and only a few studies have considered the role of anisotropic crystal symmetry in dielectrics as a delicate way to tune electronic properties of channel materials. Here, we demonstrate a layered anisotropic dielectric, SiP2, with non-symmorphic twofold-rotational C2 symmetry as a gate medium which can break the original threefold-rotational C3 symmetry of MoS2 to achieve unexpected linearly-polarized photoluminescence and anisotropic second harmonic generation at SiP2/MoS2 interfaces. In contrast to the isotropic behavior of pristine MoS2, a large conductance anisotropy with an anisotropy index up to 1000 can be achieved and modulated in SiP2-gated MoS2 transistors. Theoretical calculations reveal that the anisotropic moiré potential at such interfaces is responsible for the giant anisotropic conductance and optical response. Our results provide a strategy for generating exotic functionalities at dielectric/semiconductor interfaces via symmetry engineering.