Although gravitational radiation causes inspiralling compact binaries to circularize, a variety of astrophysical scenarios suggest that binaries might have small but nonnegligible orbital eccentricities when they enter the low-frequency bands of ground-and space-based gravitational-wave detectors. If not accounted for, even a small orbital eccentricity can cause a potentially significant systematic error in the mass parameters of an inspiralling binary [M. Favata, Phys. Rev. Lett. 112, 101101 (2014)]. Gravitational-wave search templates typically rely on the quasi-circular approximation, which provides relatively simple expressions for the gravitational-wave phase to 3.5 post-Newtonian (PN) order. Damour, Gopakumar, Iyer, and others have developed an elegant but complex quasiKeplerian formalism for describing the post-Newtonian corrections to the orbits and waveforms of inspiralling binaries with any eccentricity. Here we specialize the quasi-Keplerian formalism to binaries with low eccentricity. In this limit the non-periodic contribution to the gravitational-wave phasing can be expressed explicitly as simple functions of frequency or time, with little additional complexity beyond the well-known formulas for circular binaries. These eccentric phase corrections are computed to 3PN order and to leading order in the eccentricity for the standard PN approximants. For a variety of systems these eccentricity corrections cause significant corrections to the number of gravitational wave cycles that sweep through a detector's frequency band. This is evaluated using several measures, including a modification of the useful cycles. By comparing to numerical solutions valid for any eccentricity, we find that our analytic solutions are valid up to e0 0.1 for comparable mass systems, where e0 is the eccentricity when the source enters the detector band. We also evaluate the role of periodic terms that enter the phasing and discuss how they can be incorporated into some of the PN approximants. While the eccentric extension of the PN approximants is our main objective, this work collects a variety of results that may be of interest to others modeling eccentric relativistic binaries. This includes a consistent eccentricity expansion of the Newtonian-order polarizations and a comparison of quasi-Keplerian results with numerical simulations. In addition to applications in gravitational wave data analysis, the formulas derived here could be of use in comparing PN theory with numerical relativity or self-force calculations of eccentric binaries. They could also be useful in the construction of phenomenological inspiral-merger-ringdown waveforms that include eccentricity effects.