We investigate the quantum robustness of the topological order in the toric code on the honeycomb lattice in the presence of a uniform parallel field. For a field in zz-direction, the low-energy physics is in the flux-free sector and can be mapped to the transverse-field Ising model on the honeycomb lattice. One finds a second-order quantum phase transition in the 3D Ising^\star⋆ universality class for both signs of the field. The same is true for a postive field in xx-direction where an analogue mapping in the charge-free sector yields a ferromagnetic transverse-field Ising model on the triangular lattice and the phase transition is still 3D Ising^\star⋆. In contrast, for negative xx-field, the charge-free sector is mapped to the highly frustrated antiferromagnetic transverse-field Ising model on the triangular lattice which is known to host a quantum phase transition in the 3D XY^\star⋆ universality class. Further, the charge-free sector does not always contain the low-energy physics for negative xx-fields and a first-order phase transition to the polarized phase in the charge-full sector takes place at larger negative field values. We quantify the location of this transition by comparing quantum Monte Carlo simulations and high-field series expansions. The full extension of the topological phase in the presence of xx- and zz-fields is determined by perturbative linked-cluster expansions using a full graph decomposition. Extrapolating the high-order series of the charge and the flux gap allows to estimate critical exponents of the gap closing. This analysis indicates that the topological order breaks down by critical lines of 3D Ising^\star⋆ and 3D XY^\star⋆ type with interesting potential multi-critical crossing points. All findings for the toric code on the honeycomb lattice can be transferred exactly to the toric code on the triangular lattice.