The phenomenon mode instability is the most limiting factor for further scaling the output power and beam quality in high power fiber lasers. Thus, it is meaningful and necessary to study the influencing factor of mode instability and finally find the approaches to mitigating its influence. Theoretical calculations reveal that the fiber V-parameter has a negative effect on fiber amplifier mode instability threshold. Nevertheless, the influence of fiber core numerical aperture (<i>NA</i>) on fiber oscillator mode instability threshold has rarely been investigated compared with that on the fiber amplifier. In this paper, we build a high-power all-fiber laser oscillator pumped by 976nm laser diodes and measure its laser efficiency and mode instability threshold of 20/400 step-index ytterbium doped fiber with different fiber core <i>NA</i>. Experimental result reveals that at the same 976 nm pump power, the fiber with relatively low core <i>NA</i> (~0.059) has a higher mode instability threshold power than that with relatively high core <i>NA</i> (~0.064), and that even a higher core <i>NA</i> (~0.064) fiber has a higher laser efficiency than lower core <i>NA</i> (~0.059) fiber. The fact shows that the fiber core <i>NA</i> has a significant influence on mode instability threshold, and a relatively high core <i>NA</i> results in a lower mode instability threshold. Also, numerical simulations explain the reason why the fiber core <i>NA</i> has a negative effect on mode instability threshold in fiber oscillator. First of all, the higher fiber core <i>NA</i> will support more propagating modes in fiber, and the lower fiber core <i>NA</i> will result in higher order mode (HOM) content leaking into fiber cladding and the overlap of HOM content and gain area is reduced, thus the gain of HOM is relatively reduced. Also, the bending loss of HOM is very sensitive to fiber core <i>NA</i> variation, and the increase of fiber core <i>NA</i> will reduce the bending loss of HOM dramatically. In conclusion, the fiber core <i>NA</i> has a significant negative effect on fiber oscillator mode instability threshold, and numerical simulationscan explain the physical origin of the negative effect of fiber core <i>NA</i> on laser oscillator mode instability threshold. Thus, for the mode instability mitigation in high power laser oscillator, optimizing the <i>NA</i> of active fiber conduces to the increase of mode instability threshold, which is helpful and necessary for further scaling the output power and beam quality.