Quantitative phase imaging (QPI) is a label‐free computational imaging technique that provides optical path length information of specimens. In modern implementations, the quantitative phase image of an object is reconstructed digitally through numerical methods running in a computer, often using iterative algorithms. Here, a diffractive QPI network that can perform all‐optical phase recovery is demonstrated, and the quantitative phase image of an object is synthesized by converting the input phase information of a scene into intensity variations at the output plane. A diffractive QPI network is a specialized all‐optical processor designed to perform a quantitative phase‐to‐intensity transformation through passive diffractive surfaces that are spatially engineered using deep learning and image data. Forming a compact, all‐optical network that axially extends only ≈200–300λ, where λ is the illumination wavelength, this framework can replace traditional QPI systems and related digital computational burden with a set of passive transmissive layers. All‐optical diffractive QPI networks can potentially enable power‐efficient, high frame‐rate, and compact phase imaging systems that might be useful for various applications, including, e.g., microscopy and sensing.