The article presents the study of the formation and the analysis of the structure and properties of nanomaterials, which are cobalt-iron-niobium thin-film systems. A mathematical model is presented that allows to describe the combined behavior of atomic spins and coordinates. The generalized version of the model can use any type of the force interaction potential and take into account various contributions to the magnetic Hamiltonian depending on the specific problem being solved. When conducting computational experiments, the force potential of the modified embedded atom method (MEAM) was considered; to take into account magnetic interactions, a simplified magnetic Hamiltonian that includes the exchange interaction and the Zeeman interaction was used. The implementation of the mathematical model is presented in the LAMMPS software package, which was used to carry out numerical calculations. During the computational experiments, two separate problems were considered. In the first problem, the processes of deposition, interaction and ordering of a multilayer atomistic cobalt-iron-niobium system were analyzed. It has been shown that when niobium atoms are deposited onto a substrate, a relief nanofilm with irregularities up to a nanometer in height is formed. The contact between iron and niobium nanofilms is more blurred compared to the similar area of the contact between cobalt and iron nanofilms. In the second problem, the self-organization and reorientation of atomic spins in an iron nanofilm with a crystalline structure were analyzed. Calculations were carried out with and without an existing external magnetic field. The appearance of skyrmions (vortex flows) during the spin interaction of iron atoms was recorded. The simulation has shown that under the influence of an external magnetic field, an induced magnetic moment appears in the iron nanofilm, opposite to the induction vector of the external field.