The mechanism of X-ray outbursts in Be X-ray binaries remains a mystery, and understanding their circumstellar disks is crucial for a solution of the mass-transfer problem. In particular, it is important to identify the Be star activities (e.g., pulsations) that cause mass ejection and, hence, disk formation. Therefore, we investigated the relationship between optical flux oscillations and the infrared (IR) excess in a sample of five Be X-ray binaries. Applying the Lomb-Scargle technique to high-cadence optical light curves from the Transiting Exoplanet Survey Satellite (TESS), we detected several significant oscillation modes in the 3 to 24 hour period range for each source. We also measured the IR excess (a proxy for disk growth) of those five sources, using J-band light curves from Palomar Gattini-IR. In four of the five sources, we found anti-correlations between the IR excess and the amplitude of the main flux oscillation modes. This result is inconsistent with the conventional idea that non-radial pulsations drive mass ejections. We propose an alternative scenario where internal temperature variations in the Be star cause transitions between pulsation-active and mass-ejection-active states.