The Cherenkov electromagnetic instability excited by an oscillating relativistic electron beam in ion channel has been studied. It is shown that the complex boundary with beam-ion channel-plasma system and the perturbation of electrons induce several electromagnetic-electrostatic hybrid slow waves, which offer the possibility to excite the instability. Under such conditions, the electronic betatron motion plays a decisive role in exciting the instability. The necessary and sufficient conditions inducing the instability have been given, and related physical mechanisms have been discussed. Furthermore, the influence of the parameters, such as accelerating voltage, on the width of instability band and the growth rate of the instability has been given.