The emergence of vortex-bright soliton dipoles in two-component Bose-Einstein condensates through bifurcations from suitable eigenstates of the underlying linear system is examined. These dipoles can have their bright solitary structures be in phase (symmetric) or out of phase (antisymmetric). The dynamical robustness of each of these two possibilities is considered and the out-of-phase case is found to exhibit an intriguing symmetry-breaking instability that can in turn lead to tunneling of the bright wavefunction between the two vortex "wells". We interpret this phenomenon by virtue of a vortex-induced double well system, whose spontaneous symmetry breaking leads to asymmetric vortex-bright dipoles, in addition to the symmetric and anti-symmetric ones. The theoretical prediction of these states is corroborated by detailed numerical computations.arXiv:1207.0386v2 [cond-mat.quant-gas] 19 Oct 2012 2