Solitons and cavitons (localized solutions with singularities) for the nonlocal Whitham equations are studied. The equation of a fourth order with a parameter in front of fourth derivative for traveling waves is reduced to a reversible Hamiltonian system defined on a twosheeted four-dimensional space. When this parameter is small we get a slow-fast Hamiltonian system. Solutions of the system which stay on one sheet represent smooth solutions of the equation but those which perform transitions through the branching plane represent solutions with jumps. They correspond to solutions with singularities -breaks of the first and third derivatives but continuous even derivatives. The system has two types of equilibria on different sheets, they can of saddle-center or saddle-foci. Using analytic and numerical methods we found many types of homoclinic (and periodic as well) orbits to these equilibria both with a monotone asymptotics and oscillating ones. They correspond to solitons and cavitons of the initial equation. When we deal with homoclinic orbits to a saddle-center the values of the second parameter (physical wave speed) is discrete but for the case of a saddle-center it is continuous. The presence of majority such solutions displays the very complicated dynamics of the system.