With large-scale wind farms connected to AC/DC network, the transient stability assessment (TSA) of the power system becomes more and more difficult. Among them, estimating the region of attraction (ROA) of the equilibrium point is a traditional but still challenging problem. Based on the Lyapunov stability theory, this paper proposes a new approach to obtain the enlarged estimation of the ROA. The optimization and updating strategy of shape function in the sum of squares (SOS) optimization problem are studied to reduce the conservatism of estimation result. In the proposed method, the gravitational search algorithm (GSA) is employed to optimize the coefficients of initial shape function to improve estimation performance. Based on the time-domain simulation (TDS) of expected faults, the fitness value for shape function optimization is calculated using the values of state variables at the fault clearing time and the system stability information. Furthermore, the optimal Lyapunov function is computed by introducing the update condition of shape function and adjusting the iteration strategy of the ROA estimation algorithm. Finally, the proposed method is applied to a two-machine-infinite-bus system and a more complex nine-bus AC/DC system with wind power. And the effectiveness of the proposed method is verified by comparing with the existing ROA estimation methods.