Pupil localization is the key step of eye tracking, which directly affecting the precision of eye tracking. Through the process of eye tracking, the shape of pupil is usually distorted, leading to the invalidation of traditional algorithm based on circle localization. Moreover, current algorithms of distorted pupil localization are easily interfered by eyelid and faculae. To solve these problems, this paper raised an operator of Ellipse Difference (ED) to avoid the interference of eyelid and faculae. Meanwhile, to overcome the difficulty of ED operator's high order of complexity, this paper associated the algorithm of Particle Swarm Optimization (PSO) with ED to realize fast pupil localization. The experimental results have demonstrated ED-PSO is more robust than current algorithms and can realize real time processing, which has been applied into real eye tracking system.