We propose a multi-derivative method to reconstruct the phase of transparent objects in off-axis quantitative phase imaging (QPI). By numerically computing first-, second-, and third-order derivatives of the interferogram, we demonstrate that one can extract the quantitative phase information in a straightforward way, without prior knowledge of the carrier frequencies or Fourier transform. In contrast to existing advanced derivative methods, our approach markedly streamlines the alignment and retrieval processes, all without requiring any special prerequisites. This enhancement seamlessly translates into improved reconstruction quality. Furthermore, when compared to cutting-edge Fourier-division-based methods, our technique distinctly accelerates the phase retrieval speed. We verified our method using white-light diffraction phase microscopy and laser off-axis QPI, and the results indicate that our method can allow a fast, high-quality retrieval with frame rates up to 41.6 fps for one- megapixel interferograms on a regular computer.