TiVZrTa high-entropy alloys (HEAs) have been experimentally proven to exhibit excellent irradiation tolerance. In this work, defect energies and evolution were studied to reveal the underlying mechanisms of the excellent irradiation tolerance in TiVZrTa HEA via molecular statics calculations and molecular dynamics simulations. The atomic size mismatch of TiVZrTa is ∼6%, suggesting a larger lattice distortion compared to most face-centered cubic and body-centered cubic M/HEAs. Compared to pure Ta and V, smaller vacancy formation and migration energies with large energy spreads lead to higher equilibrium vacancy concentration and faster vacancy diffusion via low-energy migration paths. Vacancies in TiVZrTa have weaker abilities to form large vacancy clusters and prefer to form small clusters, indicating excellent resistance to radiation swelling. The formation energies of different types of dumbbells in TiVZrTa show significant differences and have large energy spreads. The binding abilities of interstitials in TiVZrTa are weaker compared to that in pure Ta and V. In TiVZrTa, fast vacancy diffusion and slow interstitial diffusion result in closer mobilities of vacancies and interstitials, significantly promoting point defect recombination. We further studied the effects of short-range ordered structures (SROs) on defect diffusion and evolution. SROs in TiVZrTa can effectively lead to higher fractions of defect recombination and fewer surviving defects. Our findings provide a comprehensive understanding of the underlying mechanisms of the high irradiation tolerance in body-centered cubic HEAs with large lattice distortion and suggest SROs are beneficial microstructures for enhancing irradiation tolerance.