Refereeing in handball is a complex exercise involving numerous decisions and interactions among players and coaches. To meet the demands of officiating a game, (e.g., withstand the pressure generated by spectators or managing critical situations), referees need to have a certain personality type. Since the sport of handball faces an acute shortage of referees and the recruitment of suitable individuals becomes increasingly difficult, personality research can provide practical guidance for the identification of potential candidates. When determining the personality characteristics necessary for refereeing, taking an external perspective and collecting information from others can provide a more objective assessment of an individual’s personality than self-assessment. For this reason, an online qualitative survey was conducted to collect data from 89 German handball players and 35 coaches. Based on the principles of qualitative content analysis, a large part of the responses could first be structured and assigned to the five-factor model. The remaining responses were then categorised, which allowed four categories to emerge: (i) character and leadership, (ii) judgement and decision-making, (iii) game management, and (iv) physical fitness and cognitive demands. The findings contribute to expanding knowledge about the crucial personality characteristics that shape handball referees’ personality and suggest that the performance profile of handball referees shows similarities to that of other team sports officials. In our conclusion, we outline research directions with high practical relevance for the recruitment and development of referees, e.g., the study of referees’ assertiveness, need for evaluation, preference for consistency, or self-efficacy.