This work presents a longitudinal study of diversity among the Affective Computing research community members. We explore several dimensions of diversity, including gender, geography, institutional types of affiliations and selected combinations of dimensions. We cover the last 10 years of the IEEE Transactions on Affective Computing (TAFFC) journal and the International Conference on Affective Computing and Intelligent Interaction (ACII), the primary sources of publications in Affective Computing. We also present an analysis of diversity among the members of the Association for the Advancement of Affective Computing (AAAC). Our findings reveal a "leaky pipeline" in the field, with a low -albeit slowly increasing over the years-representation of women. They also show that academic institutions clearly dominate publications, ahead of industry and governmental centres. In terms of geography, most publications come from the USA, contributions from Latin America or Africa being almost non-existent. Lastly, we find that diversity in the characteristics of researchers (gender and geographic location) influences diversity in the topics. To conclude, we analyse initiatives that have been undertaken in other AI-related research communities to foster diversity, and recommend a set of initiatives that could be applied to the Affective Computing field to increase diversity in its different facets. The diversity data collected in this work are publicly available, ensuring strict personal data protection and governance rules.