We have investigated the cryogenic performance of Yb:YAG and Yb:YLF crystals in rod-geometry to understand the pros and cons of each material for the development of ultrafast lasers and amplifier systems. We have performed detailed spectroscopic (absorption, emission, lifetime), temperature, lasing, and thermal-lens measurements with Yb:YLF and Yb:YAG crystals under almost identical conditions. Our analysis has shown that despite the higher thermal conductivity of Yb:YAG, due to its smaller quantum defect, the peak/average temperatures reached under similar pumping conditions is lower in Yb:YLF crystals. Moreover, since the YLF host has a negative thermo-optic coefficient, that balances other positive contributions to thermal lensing, overall Yb:YLF rods possess a much weaker thermal lens than Yb:YAG under similar conditions. As a result of these benefits, we have shown that Yb:YLF rods perform better than Yb:YAG in cryogenic lasing experiments in terms of attainable power performance and laser output beam quality. In terms of gain per pass, the Yb:YAG medium is superior, however, the gain bandwidth is much broader in Yb:YLF systems that make it more suitable for ultrafast pulse laser/amplifier development. We have further shown that, the asymmetric thermal lens behavior of Yb:YLF favors laser operation in E//c axis over E//a axis. The comparison in this study has been performed in rod geometry and for Yb-doping, however, we believe that, to first order, the discussion could be extended to YAG/YLF laser systems doped with other ions (Pr, Nd, Er, Tm, Ho) and to other lasing geometries such as slab and thin-disk.