The present search seeks to customize the optical, magnetic, and structural characteristics of NiS/Fe3O4 and CoS/Fe3O4 nanocomposites in contrast to Ni-doped Fe3O4 and Co-doped Fe3O4 nanoparticles to investigate linear, nonlinear, and optoelectronic applications. In this regard, the preparation of magnetite nanoparticles (Fe3O4) doped by Ni and Co and also NiS/Fe3O4 and CoS/Fe3O4 nanocomposites was successfully carried out through the co-precipitation method. The XRD obtained results revealed the presence of a face-centered cubic structure in all samples. Moreover, the estimation of the crystalline size for the Fe3O4, Ni-doped Fe3O4, Co-doped Fe3O4 NiS/Fe3O4, and CoS/Fe3O4 nanopowders was carried out using the Debye-Scherrer formula, yielding values ranging from 8.45 nm to 10.13 nm. Furthermore, nanopowder microstructure images were taken through a field emission scanning electron microscope (FESEM), which represented that the shape of the grains was spherical and irregular. Furthermore, the samples validate the purity of the samples which revealed the presence of vibrational modes in the metal oxide bonds. An optical investigation was conducted on all samples utilizing a diffuse reflectance spectroscopy. Band gap values were estimated based on diffuse reflectance spectroscopy data through Tauc plot analysis, which yielded a range from 2.88 eV to 3.01 eV, indicating that the red shift in the band gap of Fe3O4 with all impurities except CoS/Fe3O4, provides catalytic applications of synthesized materials. Additionally, numerical calculations utilizing the Kramers-Kronig relation and Wemple-DiDomenico (WDD) model, were performed to assess the extinction coefficient (k), refractive index (n), linear susceptibility (χ^((1) )), and third-order nonlinear susceptibility (χ^((3) )) parameters on the reflectance data. Compared to other studied samples, the maximum evaluated values of χ^((3))and the density of polarizable constituents (N) were related to the Ni-doped Fe3O4 NPs. This observation suggests that the prepared nanopowders possess significant potential for utilization in both fields of linear and nonlinear optical devices.