To effectively guarantee a secure and stable operation of a smart substation, it is essential to develop a relay protection system considering the real-time online operation state evaluation and the risk assessment of that substation. In this paper, based on action data, defect data, and network message information of the system protection device (PD), a Markov model-based operation state evaluation method is firstly proposed for each device in the relay protection system (RPS). Then, the risk assessment of RPS in the smart substation is carried out by utilizing the risk transfer network. Finally, to highly verify the usefulness and the effectiveness of the proposed method, a case study of a typical 220 kV substation is provided. It follows from the case study that the developed method can achieve a better improvement for the maintenance plan of the smart substation.