Using determinant quantum Monte Carlo (d-QMC) simulations, we demonstrate that an extended Hubbard model on a bilayer honeycomb lattice has two novel quantum phase transitions. The first is a quantum phase transition between the weakly interacting gapless Dirac fermion phase and a strongly interacting fully gapped and symmetric trivial phase, which cannot be described by the standard Gross-Neveu model. The second is a quantum critical point between a quantum spin Hall insulator with spin S z conservation and the previously mentioned strongly interacting fully gapped phase. At the latter quantum critical point the single particle excitations remain gapped, while spin and charge gap both close. We argue that the first quantum phase transition is related to the Z16 classification of the topological superconductor 3 He-B phase with interactions, while the second quantum phase transition is a topological phase transition described by a bosonic O(4) nonlinear sigma model field theory with a Θ-term.IntroductionThe interplay between topology and interactions can lead to very rich new physics. For bosonic systems, it is understood that strong interactions can lead to many symmetry protected topological (SPT) phases [1,2] that are fundamentally different from the standard Mott insulator and superfluid phases. In addition to producing various topological orders, for fermionic systems strong interactions can also reduce the classification of free fermion topological insulators and superconductors [3][4][5][6][7][8][9][10][11]. That is, interactions can drive free fermion topological superconductors to a trivial phase; namely the edge states of the free fermion topological superconductor can be gapped out without degeneracy by a symmetry preserving short range interactions without going through a bulk quantum phase transition. The most famous example is the 3 He-B topological superconductor protected by time-reversal symmetry, whose boundary is described by a (2 + 1)d Majorana fermion χ with theHe-B has a Z classification; therefore for arbitrary copies of 3 He-B, its boundary remains gapless as long as time-reversal symmetry is preserved [12][13][14]. In other words any fermion-bilinear mass term χ a σ y χ b at the boundary would break the time-reversal symmetry. However, once interactions are turned on, the classification of 3 He-B is reduced to Z 16 ; i.e., with 16 copies of 3 He-B, its boundary can be gapped out by interactions while preserving the time-reversal symmetry [9,10]. In other words, the boundary is fully gapped by interactions with χ a σ y χ b = 0, for a, b = 1 · · · 16.Although the classification of interacting 3 He-B has been understood, the following question remains: if the interactions are tuned continuously, can there be a direct second order quantum phase transition between the weakly interacting gapless boundary and the strongly interacting fully gapped nondegenerate boundary state? Even if such a second order phase transition exists, its field theory description is unknown because the standard field ...