The gravitational instability of dark-baryonic matter systems is investigated by adopting Boltzmann-Vlasov equation in the background of f(R) gravity, the dispersion relation considering the κ-deformed Kaniadakis distribution is derived by exploiting the kinetic theory. The cases of high and low frequency perturbations are analyzed, respectively. In the high-frequency regime, as κ increases, the normalized frequency experiences a corresponding increase. Conversely, in the low-frequency regime, an increase in κ leads to a simultaneous decrease in both the growth rate and the critical wave number. Furthermore, Jeans mass limit of dark-baryonic matter systems in
f
R
gravity with κ-deformed Kaniadakis distribution is studied, it is found that the increase of T
eff
with the enlargement of κ, which leads to the increase of Jeans mass M
*. In additional, the influence of dispersion velocity ratio and density ratio of dark to baryonic matter is investigated, it is indicated that the amplification of the dispersion velocity ratio leads to the decrease of Jeans mass. It is worth emphasizing that the dark-bayonic matter system have a higher limitation of instability in f(R) gravity than that in the classical Newtonian gravity.