We study the correlation of galaxy structural properties with their location relative to the SFR-M * correlation, also known as the star formation "main sequence" (SFMS), in the CANDELS and GAMA surveys and in a semi-analytic model (SAM) of galaxy formation. We first study the distribution of median Sérsic index, effective radius, star formation rate (SFR) density and stellar mass density in the SFR-M * plane. We then define a redshift dependent main sequence and examine the medians of these quantities as a function of distance from this main sequence, both above (higher SFRs) and below (lower SFRs). Finally, we examine the distributions of distance from the main sequence in bins of these quantities. We find strong correlations between all of these galaxy structural properties and the distance from the SFMS, such that as we move from galaxies above the SFMS to those below it, we see a nearly monotonic trend towards higher median Sérsic index, smaller radius, lower SFR density, and higher stellar density. In the semi-analytic model, bulge growth is driven by mergers and disk instabilities, and is accompanied by the growth of a supermassive black hole which can regulate or quench star formation via Active Galactic Nucleus (AGN) feedback. We find that our model qualitatively reproduces the trends described above, supporting a picture in which black holes and bulges co-evolve, and AGN feedback plays a critical role in moving galaxies off of the SFMS.