We show that universal holonomic quantum computation can be achieved fault tolerantly by adiabatically deforming the gapped stabilizer Hamiltonian of the surface code, where quantum information is encoded in the degenerate ground space of the system Hamiltonian. We explicitly propose procedures to perform each logical operation, including logical state initialization, logical state measurement, logical controlled-NOT (CNOT), state injection, distillation, etc. In particular, adiabatic braiding of different types of holes on the surface leads to a topologically protected, non-Abelian geometric logical CNOT. Throughout the computation, quantum information is protected from both small perturbations and low-weight thermal excitations by a constant energy gap and is independent of the system size. Also, the Hamiltonian terms have weight at most four during the whole process. The effect of thermal error propagation is considered during the adiabatic code deformation. With the help of active error correction, this scheme is fault tolerant, in the sense that the computation time can be arbitrarily long for large-enough lattice size. It is shown that the frequency of error correction and the physical resources needed can be greatly reduced by the constant energy gap.