The t2g orbitals of an edge-shared transition-metal oxide with a honeycomb lattice structure form dispersionless electronic bands when only hopping mediated by the edge-sharing oxygens is accessible. This is due to the formation of isolated quasimolecular orbitals (QMOs) in each hexagon, introduced recently by Mazin et al. [Phys. Rev. Lett. 109, 197201 (2012)], which stabilizes a band insulating phase for t 5 2g systems. However, with help of the exact diagonalization method to treat the electron kinetics and correlations on an equal footing, we find that the QMOs are fragile against not only the spin-orbit coupling (SOC) but also the Coulomb repulsion. We show that the electronic phase of t 5 2g systems can vary from a quasimolecular band insulator to a relativistic J eff = 1/2 Mott insulator with increasing the SOC as well as the Coulomb repulsion. The different electronic phases manifest themselves in electronic excitations observed in optical conductivity and resonant inelastic x-ray scattering. Based on our calculations, we assert that the currently known Ru 3+ -and Ir 4+ -based honeycomb systems are far from the quasimolecular band insulator but rather the relativistic Mott insulator. Introduction -Physical properties of 4d and 5d transition metal (TM) compounds with nominally less than six d electrons are determined by the t 2g manifold because of a strong cubic crystal field. A strong spin-orbit coupling (SOC) causes t 2g orbitals split into effective total angular momenta j eff = 1/2 and 3/2. The relativistic electronic feature in these TM compounds has drawn much attraction recently as exotic electronic, magnetic, and topological phases have been expected, including J eff = 1/2 Mott insulator [1-5], superconductivity [6,7], topological insulator [8,9], Weyl semimetal [10,11], and spin liquid [12,13]. Among them, the research on t 5 2g systems forming a honeycomb lattice structure with edge-sharing ligands has been triggered by the possibility of a nontrivial topological phase [14,15]