In order to further knowledge regarding the factors that have most influence on the university image, a measurement model was established in this research from the perspective of society and validated using covariance structure analysis. In addition, to ascertain whether such factors and/or their degree of influence differ among stakeholders, an evaluation of measurement invariance was conducted to find out what implications there are for image formation when the perspectives of the different stakeholders are considered. Five samples of stakeholders were used (society, prospective students, current students, graduates, and companies), totaling 1760 respondents. The results showed that affective image, perception of teaching resources, and perception of graduate training significantly influence the formation of overall university image from the perspective of society and that the image structure identified from this perspective is shared by companies, although some differences are observed. However, the model needs adapting to consider the perspectives of prospective students, current students, and graduates. The results also suggested that affective issues have a major importance in image formation from the perspective of all the stakeholders considered and that it is in the cognitive aspects where differences between them become more patent. These findings constitute a valuable contribution for marketing literature as so few works have addressed the study of the university image adopting the standpoint of society or the comparison of different stakeholders. Further, they provide guidance to university managers when determining which aspects are recommendable to act upon for the projection of a favorable image to various audiences.