We study the non-adiabatic dynamics of a two-state subsystem in a bath of independent spins using the non-interacting blip approximation, and derive an exact analytic expression for the relevant memory kernel. We show that in the thermodynamic limit, when the subsystem-bath coupling is diluted (uniformly) over many (infinite) degrees of freedom, our expression reduces to known results, corresponding to the harmonic bath with an effective, temperature-dependent, spectral density function. We then proceed and study the heat current characteristics in the out-of-equilibrium spin-spin-bath model, with a two-state subsystem bridging two thermal spin-baths of different temperatures. We compare the behavior of this model to the case of a spin connecting boson baths, and demonstrate pronounced qualitative differences between the two models. Specifically, we focus on the development of the thermal diode effect, and show that the spin-spin-bath model cannot support it at weak (subsystem-bath) coupling, while in the intermediate-strong coupling regime its rectifying performance outplays the spin-boson model.