In this work we study nucleon-nucleus elastic scattering using a nonlocal, velocity-dependent optical potential. The potential parameters are determined by fitting elastic angular distributions and polarization data for nucleon scattering off a wide range of nuclei falling in the mass range $12 \leq A \leq 208$ and over the energy range 10 - 60 MeV. Our potential parameters lead to smoothly varying local equivalent potentials and, unlike previous works, the potential depths corresponding to the real volume, imaginary surface, and imaginary volume terms show systematic linear dependences on energy. In addition, for each nuclear target, we determined constant sets of geometric parameters. Including the polarization data in the fitting procedure helped in reducing the large variations in the depths of the spin-orbit term. Our best-fit angular distributions and polarization data are in very good agreement with measured data, and are either as good as the cross sections obtained with widely-used systematics or better.