Electro‐optic sampling has emerged as a new quantum technique enabling measurements of electric field fluctuations on subcycle time scales. In a second‐order nonlinear material, the fluctuations of a terahertz field are imprinted onto the polarization properties of an ultrashort probe pulse in the near infrared. The statistics of this time‐domain signal are calculated, incorporating the quantum nature of the involved electric fields right from the beginning. A microscopic quantum theory of the electro‐optic process is developed adopting an ensemble of noninteracting three‐level systems as a model for the nonlinear material. It is found that the response of the nonlinear medium can be separated into a conventional part, which is exploited also in sampling of coherent amplitudes, and quantum contributions, which are independent of the state of the terahertz input. Interactions between the three‐level systems which are mediated by terahertz vacuum fluctuations are causing this quantum response. Conditions under which the classical response serves as a good approximation of the electro‐optic process are also determined and how the statistics of the sampled terahertz field can be reconstructed from the electro‐optic signal is demonstrated. In a complementary regime, electro‐optic sampling can serve as a spectroscopic tool to study the pure quantum susceptibilities of matter.