We discuss high-order calculations in perturbative effective field theory for fermions at low energy scales. The Fermi-momentum or k F a s expansion for the ground-state energy of the dilute Fermi gas is calculated to fourth order, both in cutoff regularization and in dimensional regularization. For the case of spin one-half fermions we find from a Bayesian analysis that the expansion is well converged at this order for |k F a s | 0.5. Furthermore, we show that Padé-Borel resummations can improve the convergence for |k F a s | 1. Our results provide important constraints for nonperturbative calculations of ultracold atoms and dilute neutron matter.