This paper develops the high-order accurate entropy stable (ES) finite difference schemes for the shallow water magnetohydrodynamic (SWMHD) equations. They are built on the numerical approximation of the modified SWMHD equations with the Janhunen source term. First, the second-order accurate well-balanced semi-discrete entropy conservative (EC) schemes are constructed, satisfying the entropy identity for the given convex entropy function and preserving the steady states of the lake at rest (with zero magnetic field). The key is to match both discretizations for the fluxes and the non-flat river bed bottom and Janhunen source terms, and to find the affordable EC fluxes of the second-order EC schemes. Next, by using the second-order EC schemes as building block, high-order accurate well-balanced semi-discrete EC schemes are proposed. Then, the high-order accurate well-balanced semidiscrete ES schemes are derived by adding a suitable dissipation term to the EC scheme with the WENO reconstruction of the scaled entropy variables in order to suppress the numerical oscillations of the EC schemes. After that, the semi-discrete schemes are integrated in time by using the high-order strong stability preserving explicit Runge-Kutta schemes to obtain the fully-discrete high-order well-balanced schemes. The ES property of the Lax-Friedrichs flux is also proved and then the positivity-preserving ES schemes are studied by using the positivity-preserving flux limiter. Finally, extensive numerical tests are conducted to validate the accuracy, the well-balanced, ES and positivity-preserving properties, and the ability to capture discontinuities of our schemes.