We investigate the magnetic properties in a frustrated spin system of carbonyl iron (CI) particles before and after Ni+ ion beam irradiation. Upon increasing temperatures, the saturation magnetization exhibits an anomalous increase, which is more intense after the beam irradiation. The zero-field cooled (ZFC) magnetization data show an anomalous increase up to 300 K, regardless of the beam irradiation. After the irradiation, unlike in the unirradiated CI particles, the ZFC curve shows separated regimes, reflecting two distributions of the blocking temperature, which may be related to the particle distribution summed with two distribution functions. After the irradiation, strong interparticle interaction may be present due to the effect of dipolar interaction among CI particles doped Ni ions, leading to the enhanced magnetization. We may suggest that the anomalous magnetization behavior can be ascribed to frustration in the internal magnetic order for the unirradiated CI particles, and further interparticle interaction for the irradiated CI particles.