This work reports a computational investigation of the effect of ancillary ligands on the activity of a Rh catalyst for hydrogen evolution based on the [Cp*Rh] motif (Cp* = η5-pentamethylcyclopentadienyl). Specifically, we investigate why a bipyridyl (bpy) ligand leads to H2 generation but diphenylphosphino-based (dpp) ligands do not. We compare the full ligands to simplified models and systematically vary structural features to ascertain their effect on the reaction energy of each catalytic step. The calculations, based on density functional theory, show that the main effect on reactivity is the choice of linker atom, followed by its coordination. In particular, P stabilizes the intermediate Rh-hydride species by donating electron density to the Rh, thus inhibiting the reaction towards H2 generation. Conversely N, a more electron withdrawing center, favors H2 generation at the price of destabilizing the hydride intermediate, which cannot be isolated experimentally and makes determining the mechanism for this reaction more difficult. We also find that steric effects of bulky substituents on the main ligand scaffold can lead to large effects on the reactivity, which may be challenging to fine-tune. On the other hand, structural features like the bite angle of the bidentate ligand have a much smaller impact on reactivity. Therefore, we propose that the choice of linker atom is key for the catalytic activity of this species, which can be further fine-tuned by a proper choice of electron-directing groups on the ligand scaffold.