A LieYRep pair consists of a Lie-Yamaguti algebra and its representation. In this paper, we establish the cohomology theory of LieYRep pairs, and characterize their linear deformations by the first cohomology group. Then we introduce the notion of relative Rota-Baxter-Nijenhuis structures on LieYRep pairs and investigate their properties. As applications, we show that a relative Rota-Baxter-Nijenhuis structure gives rise to a Maurer-Cartan operator on a twilled Lie-Yamaguti algebra under certain condition, and obtain the equivalence between r-matrix-Nijenhuis structures and Rota-Baxter-Nijenhuis structures on Lie-Yamaguti algebras.