We studied the intrinsic and extrinsic Gilbert damping constants measured with an all-optical method in an exchange-biased IrMn/Cu/CoFe trilayer system with various Cu-spacer layer thicknesses. To understand the effect of the exchange bias on damping, the strength of the exchange bias was controlled by the Cu-spacer layer. When the external magnetic field is high enough compared with the exchange fields, the intrinsic Gilbert damping constant is independent of the exchange bias field. On the other hand, under a low external magnetic field, the extrinsic Gilbert damping is dominant because of the magnetic inhomogeneities, and is strongly correlated with the exchange bias field.