In order to establish the correlation between the magnetoelectric coupling and magnetic instability, we have studied the structural, magnetic, and ferroelectric properties of BaTiO modified BiCoOi.e. xBaTiO-(1 - x)BiCoO as a function of BaTiO concentration (x) and volume from a series of general-gradient-corrected (GGA), GGA plus onsite Coulomb repulsion (U), full potential, spin-density-functional band-structure calculations within the framework of density functional theory along with synchrotron X-ray diffraction and magnetic measurement studies. G-type antiferromagnetic ordering was found to be energetically favorable among all the considered magnetic configurations for x < 0.45 and higher concentrations stabilize with nonmagnetic (NM) states. We observe metamagnetic spin state transitions associated with paraelectric to ferroelectric transitions as a function of volume and x using synchrotron diffraction and computational studies, indicating a strong magnetoelectric coupling. Specifically for x = 0.33 composition, a pressure induced high spin (HS) to low spin (LS) transition occurs when the volume is compressed below 2.5%. Our orbital-projected density of states show a HS state for Co in the ferroelectric ground state for x < 0.45 and the corresponding paraelectric phase is stable in the NM state due to the stabilization of LS state as evident from our fixed-spin-moment calculations and magnetic measurements. The nature of chemical bonding has been studied using partial density of states, electron localization function, and Born effective charge analysis. High values of spontaneous ferroelectric polarizations are predicted for lower x values which inversely vary with x because of the reduction of tetragonality (c/a) with increase in x which indicates the presence of both spin-lattice and ferroelectricity-lattice coupling. Our partial polarization analysis shows that not only the lone pair at Bi sites but also the d-ness of Ti ions contribute to the net polarization. Moreover, we find that the HS-LS transition point and magnetoelectric coupling strength can be varied by x.