According to the theory of passivity-based control (PBC), this paper establishes a port-controlled Hamiltonian system with dissipation (PCHD) model for a doubly fed induction generator (DFIG) system under unbalanced grid voltage conditions and proposes a method of interconnection and damping assignment passivity-based control (IDA-PBC) of the system under such conditions. By using this method, the rotor-side converter and grid-side converter can be controlled simultaneously in order to improve fault ride-through capability of the DFIG system. Simulation results indicate that this IDA-PBC strategy effectively suppresses fluctuations of output current and power in the DFIG system during unbalanced grid voltage sag/swell, enhances dynamic performance, and improves the robustness of the system.