Gender is an important risk factor for adverse drug reactions. Women report significantly more adverse drug reactions than men. There is a growing consensus that gender differences in drug PK is a main contributor to higher drug toxicity in women. These differences stem from physiological differences (body composition, plasma protein concentrations, and liver and kidney function), drug interactions, and comorbidities. Contrast agents are widely used to enhance diagnostic performance in computed tomography and magnetic resonance imaging. Despite their broad use, these contrast agents can lead to important adverse reactions including hypersensitivity reactions, nephropathy, and hyperthyroidism. Importantly, female gender is one of the main risk factors for contrast agent toxicity. As these adverse reactions may be related to gender differences in PK, this perspective aims to describe distribution and elimination pathways of commonly used contrast agents and to critically discuss gender differences in these processes.