The accurate prediction of blade tip vortices continues to be challenging for the investigation of rotor aerodynamics. In the current work, the blade tip vortex system of a Caradonna–Tung rotor in the hovering state is simulated on account of the framework of a high-order WENO scheme and hybrid RANS/LES method progressively. With the RANS method based on a fifth-order WENO–Roe scheme, the spatial resolution of wake age and capture the accuracy of tip vortices are improved significantly. However, the unsteadiness of the vortex system fails to be distinguished. Then, the DDES and IDDES methods, coupled with a fifth-order WENO–Roe scheme, are implemented to further enhance the resolution of the spatial turbulence. Compared with RANS, the improvement of spatial resolution is reflected in both the statistical and transient results with DDES. The identifiable vortex core distributions can be predicted at older wake ages. The asymmetrical characteristic of blade tip vortices is revealed along with the release of fluctuation while the predicted distribution of velocity profiles in the vortex core is enhanced. With the application of IDDES, the secondary turbulent structures are captured around the primary vortex core of blade tip vortices while the complex behaviors of the vortices are observed. However, the statistics of the averaged flow field are similar to DDES.