This paper concerns the fast evaluation of the matvec g = Kf for K ∈ C N ×N , which is the discretization of an oscillatory integral transform g(x) = K(x, ξ)f (ξ)dξ with a kernel function K(x, ξ) = α(x, ξ)e 2πıΦ(x,ξ) , where α(x, ξ) is a smooth amplitude function , and Φ(x, ξ) is a piecewise smooth phase function with O(1) discontinuous points in x and ξ. A unified framework is proposed to compute Kf with O(N log N ) time and memory complexity via the non-uniform fast Fourier transform (NUFFT) or the butterfly factorization (BF), together with an O(N ) fast algorithm to determine whether NUFFT or BF is more suitable. This framework works for two cases: 1) explicit formulas for the amplitude and phase functions are known; 2) only indirect access of the amplitude and phase functions are available. Especially in the case of indirect access, our main contributions are: 1) an O(N log N ) algorithm for recovering the amplitude and phase functions is proposed based on a new low-rank matrix recovery algorithm; 2) a new stable and nearly optimal BF with amplitude and phase functions in a form of a low-rank factorization (IBF-MAT) is proposed to evaluate the matvec Kf . Numerical results are provided to demonstrate the effectiveness of the proposed framework.