We investigate alternative annealing schedules on the current generation of quantum annealing hardware (the D-Wave 2000Q), which includes the use of forward and reverse annealing with an intermediate pause. This work provides new insights into the inner workings of these devices (and quantum devices in general), particular into how thermal effects govern the system dynamics. We show that a pause mid-way through the anneal can cause a dramatic change in the output distribution, and we provide evidence suggesting thermalization is indeed occurring during such a pause. We demonstrate that upon pausing the system in a narrow region shortly after the minimum gap, the probability of successfully finding the ground state of the problem Hamiltonian can be increased over an order of magnitude. We relate this effect to relaxation (i.e. thermalization) after diabatic and thermal excitations that occur in the region near to the minimum gap. For a set of large-scale problems of up to 500 qubits, we demonstrate that the distribution returned from the annealer very closely matches a (classical) Boltzmann distribution of the problem Hamiltonian, albeit one with a temperature at least 1.5 times higher than the (effective) temperature of the annealer. Moreover, we show that larger problems are more likely to thermalize to a classical Boltzmann distribution. arXiv:1810.05881v2 [quant-ph]