In this paper, we report a complex magnetic behavior arising due to the interplay of three active magnetic cations (Nd/Sm, Co and Ir), forming 3d-5d-4f magnetic sublattices. The B-site ordered double perovskites Nd2CoIrO6 and Sm2CoIrO6 were successfully prepared by conventional solid-state method. Detailed structural analysis revealed that both samples crystallized in monoclinic structure with P21/n (No.-14) space group. XPS analysis confirms the presence of multiple oxidation states of Ir (i.e., Ir4+ and Ir5+). Both samples show a paramagnetic-ferrimagnetic (TFiM) transition around 100 K, additionally, a low temperature transition is observed at around 10 K in the SCIO sample. This FM-like behaviour of the samples is attributed to the antiparallel alignment of the Co2+ and Ir4+ spins, resulting in FIM ordering. The ac susceptibility analysis confirms a spin glass type transition just below the long-range ordering temperature in NCIO sample, The obtained characteristic flipping time of a single spin from both the law (Power law and Vogel-Fulcher law) and nonzero Vogel-Fulcher temperature (T0≃ 89.1 K) further verified the existence of cluster spin-glass behavior below the ordering temperature. The temperature evolution of phonon modes (up to 4 K) suggests that the phonon mode above the magnetic ordering temperature is mainly governed by the lattice degrees of freedom, notable renormalization of the mode frequency below the ordering temperature is due to the coupling of lattice with the underlying magnetic degrees of freedom.