Vortex–antivortex pairs have shown great potential in spintronics, where they can be used for information storage and logical devices. However, the physical mechanism for the nucleation of vortex–antivortex pairs is still unclear due to its metastability. We report on spontaneous nucleation of vortex–antivortex pairs in patterned Fe20Ni80 films (circular, square, hexagonal islands). By using a complex approach involving micromagnetic simulations, more in-depth understanding of vortex pair nucleation was achieved. A large amount of vortex–antivortex pairs appear in the as-grown magnetic film, which is the unstable high-energy state. Then, vortex and antivortex moves towards each other and annihilate, transforming magnetic structures and lowering the total energy of the system. With the decrease of sizes of microstructures, isolated vortex becomes stabilized due to confinement effect. These results provide a physical view for the nucleation of vortex–antivortex pairs and may be useful for design and optimization of magnetic microstructures for future spintronic applications.