Phoxonic quasicrystals possess abundant localized modes and low-index-contrast bandgaps, thus showing much advantage over periodic counterparts. We calculated the modulation of photonic modes for terahertz waves by phononic modes in defect-free phoxonic quasicrystals (FDQs) and quasicrystals having center point defects (PDQs). Both the moving interface (MI) and photoelastic effects are considered in acousto-optic coupling. Terahertz modes can be efficiently modified by acoustic modes due to the simultaneous confinement of electromagnetic and elastic fields both inside and outside the bandgaps. The acousto-optic coupling strength depends on the selection of the acoustic and terahertz defect modes. The predominant coupling mechanism is determined by the overlapping between the terahertz and acoustic modes in PDQs, while the MI effect becomes predominant in FDQs. Our results are important for terahertz wave modulation with ultra-small footprint size.