In the terahertz (THz) band, modulation research has become a focal point, with precise control of the phase shift of THz waves playing a pivotal role. In this study, we investigate the optical control of THz phase shift modulation in a polydimethylsiloxane (PDMS)-vanadium dioxide (VO2) flexible material using THz time-domain spectroscopy. Under the influence of an 808-nm continuous wave (CW) laser with power densities ranging from 0 to 2.74 W/cm2, the PDMS-VO2 flexible material exhibits significant phase shift modulation in the frequency range of 0.2 to 1.0 THz. The maximum optical-pumping phase shift reaches 0.27π rad at 1.0 THz in a composite material with a VO2 mass fraction of 5% and a thickness of 360 µm, and the amplitude transmittance from 0.2 THz to 1.0 THz exceeds 70%. Furthermore, the composite material exhibits good stability under at least 640 switching cycle times, as confirmed through repeatability tests. The proposed composite devices offer a new approach for more flexible phase shift modulation owing to the flexibility of the composite material and the non-contact and precise modulation of light control. Additionally, the stress-adjustable characteristics of flexible materials make them highly suitable for use in wearable THz modulators, highlighting their significant application potential.