In this study, we have examined the effects of transfer residue and sample annealing on the optical properties of chemical vapor deposited graphene, transferred onto a sapphire substrate. The optical absorption of graphene was obtained from point-by-point inversion of spectroscopic ellipsometry measurements in the visible and ultraviolet ranges (250–800 nm). Measured spectra were analyzed by optical models based on the Fresnel coefficient equations. The optical models were supported by correlated Raman, scanning electron microscopy, and atomic force microscopy measurements. The obtained data were phenomenologically described by a Fano model. Our results show that a residue layer left on graphene can significantly increase its optical absorption in the visible range, compared to an annealed sample.