We present a scheme to evaluate thermodynamic variables for a system coupled to a heat bath under a time-dependent external force using the quasi-static Helmholtz energy from the numerically "exact" hierarchical equations of motion (HEOM). We computed the entropy produced by a spin system strongly coupled to a non-Markovian heat bath for various temperatures. We showed that when changes to the external perturbation occurred sufficiently slowly, the system always reached thermal equilibrium. Thus, we calculated the Boltzmann entropy and the von Neumann entropy for an isothermal process, as well as various thermodynamic variables, such as changes in internal energies, heat, and work, for a system in quasi-static equilibrium based on the HEOM. We found that although the characteristic features of the system entropies in the Boltzmann and von Neumann cases as a function of the system-bath coupling strength are similar, those for the total entropy production are completely different. The total entropy production in the Boltzmann case is always positive, whereas that in the von Neumann case becomes negative if we chose a thermal equilibrium state of the total system (an unfactorized thermal equilibrium state) as the initial state. This is because the total entropy production in the von Neumann case does not properly take into account the contribution of the entropy from the system-bath interaction. Thus, the Boltzmann entropy must be used to investigate entropy production in the fully quantum regime. Finally, we examined the applicability of the Jarzynski equality.