The performance of interFoam (a widely-used solver within the popular open source CFD package OpenFOAM) in simulating the propagation of a nonlinear (stream function solution) regular wave is investigated in this work, with the aim of systematically documenting its accuracy. It is demonstrated that over time there is a tendency for surface elevations to increase, wiggles to appear in the free surface, and crest velocities to become (severely) over estimated. It is shown that increasing the temporal and spatial resolution can mitigate these undesirable effects, but that a relatively small Courant number is required. It is further demonstrated that the choice of discretization schemes and solver settings (often treated as a "black box" by users) can have a major impact on the results. This impact is documented, and it is shown that obtaining a "diffusive balance" is crucial to accurately propagate a surface wave over long distances without requiring exceedingly high temporal and spatial resolutions. Finally, the new code isoAdvector is compared to interFoam, which is demonstrated to produce comparably accurate results, while maintaining a sharper surface. It is hoped that the systematic documentation of the performance of the interFoam solver will enable its more accurate and optimal use, as well as increase awareness of potential shortcomings, by CFD researchers interested in the general CFD simulation of free surface waves.