A rising rate of suicide among the elderly in rural China has been recognized to be triggered by mental health-associated factors. This study uses 3,397 sampled rural elderly adults from China Labor-force Dynamic Survey in 2016 to explore the response mechanism through which non-agricultural employment participation by the elderly adults in rural China can influence their mental health. Utilizing the Multivariate Regression, Instrumental Variable and Propensity Score Matching methods, we find that, the rural elderly adults who participate in local non-agricultural employment significantly improve their mental health. Self-employment tends to have a greater positive contribution to the mental health of the elderly population than waged employment. Further, work income, need for belongingness and respect, and human capital development significantly mediates the influence of participation in local non-agricultural employment on the mental health of the elderly adults. Finally, we put forward relevant policy suggestions to improving the mental health of the elderly in the countryside.