Varifocal metalenses with tunable focal length allowing for transforming the complex and bulky tunable lens kit into a single flat lens, show great potential applications in tomography techniques, 3D displays, multi-imaging processing, among others. However, varifocal metalenses that integrate key properties, such as modest efficiency, large NA as well as polarization insensitivity into one remain elusive. Here, we present a generic design principle enabling binary switching of polarization-insensitive bifocal metalenses via simply switching two target wavelengths. The underlying mechanism relies on independently tailoring two sets of distinct anisotropic nanofins that are intelligently integrated into a monolayer single-celled phase-change metasurface in the form of "cross" or "straight" composite, which not only enables the crosstalks between two wavelength channels minimized, but also allows for the metalens design to be endowed polarization-insensitive and varifocal performance while maintaining high focusing efficiency and diffraction-limited performance. Due to the noninterleaved arrangement making the meta-structure more compact, wavelength-actuated varifocal metalens with ultrahigh-NAs (0.89 and 0.95, respectively corresponding to the target wavelengths of λ1 and λ2) can be implemented with a field of view of 4° based on this principle. Additionally, we also demonstrate the versatility of our approach by successfully implementing the generation of the polarization-insensitive varifocal focusing optical vortex. Our design can easily extend to visible or THz wavebands and will enable enormous applications in miniaturized imaging devices such as cell phones, wearable displays, and virtual reality (VR)/augmented reality (AR).