Based on the mean-field determination of the single-particle energy in nuclear matter that contains naturally a rearrangement term (RT) implied by the Hugenholtz-van Hove theorem, the folding model of the nucleon optical potential (OP) is extended to take into account the RT using the effective, density dependent CDM3Yn interaction. With the exchange part of the nucleon folded OP treated exactly in the Hartree-Fock manner, a compact nonlocal version of the folding model is suggested in the present work to determine explicitly the isospin-dependent, nonlocal central term of the nucleon OP. To solve the optical model (OM) equation with a complex nonlocal OP, the calculable R-matrix method is used to analyse the elastic neutron and proton scattering on 40,48 Ca, 90 Zr, and 208 Pb targets at low energies. The inclusion of the RT into the folding model calculation of the nonlocal nucleon OP was shown to be essential for the overall good OM description of elastic nucleon scattering. To validate the nonlocal version of the folding model, the OM results given by the nonlocal folded nucleon OP are also compared with those given by the global parametrization of the nonlocal OP using the analytical nonlocal form factor suggested by Perey and Buck.