In this study, the impact of the type of ligand at the surface of colloidal CdSe@CdS dot-in-rod nanostructures on the basic exciton relaxation and charge localization processes is closely examined. These systems have been introduced into the field of artificial photosynthesis as potent photosensitizers in assemblies for light driven hydrogen generation. Following photoinduced exciton generation, electrons can be transferred to catalytic reaction centers while holes localize into the CdSe seed, which can prevent charge recombination and lead to the formation of long-lived charge separation in assemblies containing catalytic reaction centers. These processes are in competition with trapping processes of charges at surface defect sites. The density and type of surface defects strongly depend on the type of ligand used. Here we report on a systematic steady-state and time-resolved spectroscopic investigation of the impact of the type of anchoring group (phosphine oxide, thiols, dithiols, amines) and the bulkiness of the ligand (alkyl chains vs. poly(ethylene glycol) (PEG)) to unravel trapping pathways and localization efficiencies. We show that the introduction of the widely used thiol ligands leads to an increase of hole traps at the surface compared to trioctylphosphine oxide (TOPO) capped rods, which prevent hole localization in the CdSe core. On the other hand, steric restrictions, e.g., in dithiolates or with bulky side chains (PEG), decrease the surface coverage, and increase the density of electron trap states, impacting the recombination dynamics at the ns timescale. The amines in poly(ethylene imine) (PEI) on the other hand can saturate and remove surface traps to a wide extent. Implications for catalysis are discussed.