We introduce a generalized two-dimensional orbital compass model, which
interpolates continuously from the classical Ising model to the orbital compass
model with frustrated quantum interactions, and investigate it using the
multiscale entanglement renormalization ansatz (MERA). The results demonstrate
that increasing frustration of exchange interactions triggers a second order
quantum phase transition to a degenerate symmetry broken state which minimizes
one of the interactions in the orbital compass model. Using boson expansion
within the spin-wave theory we unravel the physical mechanism of the symmetry
breaking transition as promoted by weak quantum fluctuations and explain why
this transition occurs only surprisingly close to the maximally frustrated
interactions of the orbital compass model. The spin waves remain gapful at the
critical point, and both the boson expansion and MERA do not find any
algebraically decaying spin-spin correlations in the critical ground state.Comment: 9 pages, 6 figures, improved presentation, version to appear in Phys.
Rev.