This paper offers novel non‐linear control of a doubly fed induction generator (DFIG) and unified power flow controller (UPFC) for transient stability increment with transient energy function (TEF) and sliding mode observer in a power system. First, the TEF technique is considered for the damping control in a power system with a synchronous generator (SG), DFIG, and UPFC, and then the controller time‐derivative signals are estimated by a second‐order sliding mode observer. The importance of the issue is in the employ of a complete UPFC model. Also, a one‐axis model is used for DFIG that is similar to the SG model. Another characteristic of the novel non‐linear technique is robust versus system topology changes and variable time delay of the control signals. To evaluate the performance of the novel non‐linear control method for increment transient stability, simulation studies are performed on a two‐machine connected to an infinite bus (TMIB), the IEEE 9‐bus, and the New England Standard 39‐bus power system. The results determine that the novel non‐linear control method decreases the first swing of fluctuations admissibly and enhancement stability margins considerably.