We report the findings of an extensive and systematic study on the effect of crystal grain size on the response of field-driven disordered ferromagnetic systems with thin, intermediate, and bulk geometry. For numerical modeling we used the athermal nonequilibrium variant of the random field Ising model simulating the systems with tightly packed and uniformly cubic-shaped, magnetically exchange-coupled crystal grains, conducted over a wide range of grain sizes. Together with the standard hysteresis loop characterizations, we offer an in-depth examination of the avalanching response of the system, estimating the effective grain-size-related exponents by analyses of the distributions of various avalanche parameters, average avalanche shape and size, and power spectra. Our results demonstrate that grain size plays an important role in the behavior of the system, outweighing the effect of its geometry. For sufficiently small grains, the characteristics of the system response are largely unaffected by grain size; however, for larger grains, the effects become more noticeable and show up as distinct asymmetry in the magnetization susceptibilities and average avalanche shapes, as well as characteristic kinks in the distributions of avalanche parameters, susceptibilities, and magnetizations for the largest grain sizes. Our insights, unveiling the sensitivity of the system’s response to the underlying structure in terms of crystal grain size, may prove beneficial in interpreting and analyzing experimental results obtained from driven disordered ferromagnetic samples of different geometries, as well as in extending the range of possible applications.