Ferromagnetic resonance (FMR) linewidth (LW) is a tool for studying the high frequency properties of magnetic materials for their application in high-speed devices. Here, we investigate different mechanisms which determine FMR damping in bilayer ferromagnetic/antiferromagnetic (F/AF and AF/F) exchange bias systems. Variations of FMR LW with the thickness and deposition order of the F and AF layers were studied, as well as their correlation with the exchange bias field and roughness of the sample surface. We observed much larger LW in AF/F structures compared with F/AF samples. It was found that neither the exchange bias nor surface/interface roughness in the samples could explain the difference in LW for F/AF and AF/F samples. Instead, the different underlayer microstructure influenced the grainsize, leading to different angular dispersion of magnetization and different internal stray field in F-layers, promoting a different intensity of magnon scattering and FMR damping in F/AF and AF/F samples.