Ferromagnetism in van der Waals systems, preserved down to a monolayer limit, attracted attention to a class of materials with general composition CrX3 (X=I, Br, and Cl), which are treated now as canonical 2D ferromagnets. Their diverse magnetic properties, such as different easy axes or varying and controllable character of in‐plane or interlayer ferromagnetic coupling, make them promising candidates for spintronic, photonic, optoelectronic, and other applications. Still, significantly different magneto‐optical properties between the three materials have been presenting a challenging puzzle for researchers over the last few years. Herewith, it is demonstrated that despite similar structural and magnetic configurations, the coupling between excitons and magnetization is qualitatively different in CrBr3 and CrI3 films. Through a combination of the optical spin pumping experiments with the state‐of‐the‐art theory describing bound excitonic states in the presence of magnetization, we concluded that the hole‐magnetization coupling has the opposite sign in CrBr3 and CrI3 and also between the ground and excited exciton state. Consequently, efficient spin pumping capabilities are demonstrated in CrBr3 driven by magnetization via spin‐dependent absorption, and the different origins of the magnetic hysteresis in CrBr3 and CrI3 are unraveled.