We consider the Ruderman-Kittel-Kasuya-Yosida (RKKY) interaction between magnetic impurities on the surface of a three-dimensional topological insulator with proximity induced superconductivity. A superconductor placed on the top of the topological insulator induces a gap in the surface electron states and gives rise to a long-ranged in-plane antiferromagnetic RKKY interaction. This interaction is frustrated due to strong spin-orbit coupling, decays as 1/r for r < ξ, where r is the distance between two magnetic impurities and ξ the superconducting coherence length, and dominates over the ferromagnetic and Dzyaloshinskii-Moriya type interactions for r > ξ. We find the condition for the Yu-Shiba-Rusinov intragap states that are bound to the magnetic impurities. Motivated by the intense research on the interplay of topological insulators (TI), magnetism, [10][11][12][13][14][15][16][17][18], and superconductivity [1,19,20], we study here the RKKY interaction between magnetic impurities (localized spins) on the surface of a three-dimensional TI with proximity induced superconductivity, see Fig. 1. The effect of strong spin-orbit interaction at the surface of a TI on the RKKY interaction without superconductivity is well-studied theoretically [10][11][12][13][14][15]. If the system is at the charge neutrality point, i.e., the chemical potential is at the Dirac point, it is predicted that the localized spins can form a ferromagnetic order with magnetization pointing normal to the surface of the TI, if the out-of-plane anisotropy of the RKKY interaction is stronger than the in-plane one. Out-of-plane ferromagnetic order breaks time-reversal symmetry and opens energy band gaps in the surface states. On the other hand, the in-plane interaction is frustrated. It may result in a spin-glass state [21,22] if the in-plane anisotropy is stronger than the out-of plane one [14].The effect of magnetic impurities deposited on the surface of a TI on the band gap and the resulting magnetic order is under experimental debate [23][24][25][26][27][28][29][30]. Alternatively, the locally induced electronic spin density can be measured by NV-center atomic force microscope tips, allowing one to experimentally probe the spin-spin correlations independently of the magnetic impurities on the surface [31]. The method of spin-polarized scanning tunneling microscope for probing the surface magnetic properties of TI is well established experimentally [30].In the present paper, we consider an s-wave superconductor placed on the surface of a three-dimensional TI. The proximity effect can lead to a gap in the density