Rural runoff with abundant nutrients has become a great threat to aquatic environment. Hence, more and more attention has been focused on nutrients removal. In this study, an improved aerobic/anaerobic/aerobic three-stage water spinach constructed wetland (O-A-O-CW) was used to improve the removal of nitrogen and phosphorus of rural runoff. The removal rate of the target pollutants in O-A-O-CW was compared with the common matrix flow wetland as well as the no-plant wetland. The results showed that the O-A-O-CW significantly increased the chemical oxygen demand, total phosphorus, ammonium-nitrogen, nitrate, and total nitrogen removal rate, and the corresponding removal rate was 55.85%, 81.70%, 76.64%, 89.78%, and 67.68%, respectively. Moreover, the best hydraulic condition of the wetland, including hydraulic retention time and hydraulic loading, was determined, which were 2 days and 0.45 m3·m−2·day−1, respectively. Furthermore, the removal mechanism of the constructed wetland was thoroughly studied, which included the adsorption of nitrogen and phosphorus by the matrix and water spinach, and the nitrification and denitrification by the bacteria. The results demonstrated that the mechanisms of nitrogen removal in the new type wetland were principally by the nitrification and denitrification process. Additionally, adsorption and precipitation by the matrix are mainly responsible for phosphorus removal. These results suggested that the new O-A-O-CW can efficiently removal nutrients and enhance the water quality of the rural runoff.