To solve the three-phase unbalance and reactive power problems in the electric railway power supply system, this paper proposes a Scott transformer based hypotenuse hybrid traction power system (Scott-HHPS). Compared with other co-phase traction power supply systems, the Scott-HHPS could achieve the same compensation objective with smaller active compensation capacity. The paper analyzes the basic principle and makes a comparison between the Scott-HHPS and typical co-phase power supply system. The mathematical relationships among the load current, power factor and capacity demand in condition of power factor variation and regenerative braking are analyzed. In order to further reduce the compensation capacity of the Scott-HHPS, the partial compensation strategy in respect to the index of power factor and unbalance is proposed. Besides, an instantaneous current detection method based on the theorem of conservation of power is adopted to simplify the control algorithms. Simulations and experiments are both constructed and the results demonstrate that the Scott-HHPS can effectively improve the power quality in the electric railway power supply system.