The existing studies of the temperature effect of the spin-boson model show that the tunneling splitting will increase with temperature, no matter how weak the couplings are between the bath and the spin. However, the small polaron theory said that in the weak coupling and low temperature regime, the tunneling is, in fact, dominated by the diagonal transitions whereas this diagonal contribution to the tunneling should be faded away with increasing temperature. Taking advantage of the analysis originated from the Feynman-Vernon's influence functional theory, the influence on the tunneling by the phonon bath can be expressed as a product of the effects of every single phonon mode, which can be studied by numerical exact diagonalization. We find that, in the weak coupling and low temperature regime, all the spin-single-mode systems exhibit the same decreasing dependence of tunneling on increasing temperature. In turn, with the conclusion of influence functional theory, the tunneling of the spin-boson model should decrease and it is independent of the bath structure. In the strong coupling regime, however, the temperature effect reverses from suppressing to enhancing the tunneling with the increase of temperature. Discrepancies between the old theories and the small polaron theory are also explained.