Riserless mud-recovery (RMR) drilling technology was widely applied in recent years. Compared with traditional deepwater drilling, RMR drilling can improve drilling efficiency, reduce risks, and minimize environmental effects. This paper focuses primarily on the stability of a mud-return circulation system in an RMR system. First, various factors that affect the stability of a mud-return circulation system are analyzed. An analytical model for the skid-and-mud-return line is established. Second, relevant data are derived from theoretical calculations and experiments. ABAQUS software is used to analyze the effects of each factor on the stability of the mud-return circulation system. The influencing patterns of each factor on the stability of the mud-return circulation system are summarized. Furthermore, the stability of the system under different operating conditions is analyzed based on the coupling of multiple factors. The support vector regression with derivative significance weight analysis (SVR-DWSA) algorithm is employed to perform a weight analysis of the effect on the system’s stability. Finally, based on the research findings on the stability of the mud-return circulation system, relevant conclusions and recommendations are drawn. The results of this study provide valuable references for the application of RMR technology.