It is well-known that the primordial scalar curvature and tensor perturbations, ζ and γij, are conserved on super-horizon scales in minimal inflation models. However, their wave functional has a rapidly oscillating phase which is slow-roll unsuppressed, as can be seen either from boundary (total-derivative) terms of cosmological perturbations, or the WKB approximation of the Wheeler-DeWitt equation. Such an oscillatory phase involves gravitational non-linearity between scalar and tensor perturbations. By tracing out unobserved modes, the oscillatory phase causes faster decoherence of primordial gravitons compared to those by bulk interactions. Our results put a stronger lower bound of decoherence effect to the recent proposals probing squeezed primordial gravitons.