Endothelial cell apoptosis plays an important role in the pathophysiological mechanisms of vascular complications in diabetes mellitus (DM). NADPH oxidase 4 (Nox4)-dependent reactive oxygen species (ROS) aggregation is the main cause of vascular endothelial cell apoptosis. The transforming growth factor-β1 (TGF-β1)/Smad2 signaling pathway is involved in the apoptosis of several types of cells. However, the association between vascular endothelial cell apoptosis and Nox4, and the involvement of the TGF-β1/Smad2 signaling pathway in vascular endothelial cell apoptosis remain unclear. In the present study, we aimed to investigate the role of Nox4-dependent ROS production and to determine the involvement of the TGF-β1/Smad2 signaling pathway in endothelial cell apoptosis induced by oxidative stress which causes vascular injury in DM. We demonstrated that hydrogen peroxide (H2O2) increased Nox4-dependent-ROS aggregation, as well as the expression of TGF-β1, Smad2, Bax and caspase-3, decreased Bcl-2 expression and increased the apoptosis of human umbilical vein endothelial cells (HUVECs). Treatment with diphenyliodonium (DPI), a specific inhibitor of Nox4 or astragaloside IV (AST IV), a monomer located in an extract of astragaloside, decreased Nox4 expression and the levels of ROS, decreased TGF-β1 and Smad2 expression, altered the expression of apoptosis-related genes and decreased the apoptosis of HUVECs. Treatment with LY2109761, a selective inhibitor of the TGF-β1/Smad2 pathway, produced results similar to those of DPI; however, LY2109761 had no effect on Nox4 expression and ROS levels. Taken together, the findings of the present study suggest that H2O2 contributes to HUVEC apoptosis by inducing Nox4-dependent ROS aggregation and activating the TGF-β1/Smad2 signaling pathway. Our data indicate that the protective effects of AST IV against vascular endothelial cell apoptosis in DM are mainly associated with the decrease in Nox4 expression through the TGF-β1/Smad2 signaling pathway. Furthermore, the inhibition of the activation of the TGF-β1/Smad2 signaling pathway may be another potential therapeutic strategy in the treatment of DM.