The effects of the complex seismic structure in the lowermost mantle on the seismic SmKS phases that propagate beneath the core-mantle boundary are important, but as yet unclear. Thus, in this study, broadband waveform modeling with the spectral element method is conducted using the Earth Simulator. One-hour length seismograms are first synthesized with one-dimesional velocity structure of PREM, and the portions of the SmKS phases are retrieved. The shortest period that the Earth Simulator can achieve is up to 3.5 s, which is too long to reproduce S5KS and higher SmKS phases. To read the differential travel times of SmKS phases accurately, the phase-weighted stack is adopted and the uncertainty is inferred with the bootstrap method. Next, wave fields are simulated with three-dimensional velocity structures of S20RTS with emphasized velocity perturbation at the base of the mantle and SB4L18 expanded by spherical harmonics. The Earth Simulator enables the generation of a three-dimensional (3D) structure using spherical harmonics coefficients of up to 40 degrees. The different models result in different residuals for differential travel times of S4KS-S3KS and S3KS-S2KS and change in the incident azimuths of S3KS with respect to S2KS, even if global tomography models with longwavelength heterogeneity of several thousand kilometers are used. These results clearly suggest that there are strong effects of heterogeneity in the lowermost mantle on the differential travel times of S4KS-S3KS and S3KS-S2KS. The characteristcs of the uncertainty depend on the 3D-mantle models, which may provide clues to the separation of the effects of heterogeneity at the base of the mantle for SmKS anomalies.