In various team sports, such as handball, referees work on the court by continuously moving with the players. Therefore, their physical fitness also has an impact on their reaction time, which could affect their professional decisions. The cardiorespiratory fitness status of healthy Hungarian elite handball referees was examined via body composition analysis and vita maxima cardiopulmonary exercise testing with lactate measurements. One hundred referees were examined (age: 29.0 ± 7.9 years; male: 64.0%; training: 4.3 ± 2.0 hours/week; ratio of former elite handball players: 39.0%; 51.0% first and 49.0% second division referees of the Hungarian National Handball Leagues). A resting heart rate (HR) of 79.0 ± 12.6 BPM was measured. On the basis of the body composition analysis the fat-free mass index proved to be 19.9 ± 2.6 kg/m2. The referees achieved a maximal oxygen uptake (V̇O2max) of 44.6 ± 6.1 ml/kg/min, with a maximal HR of 187.2 ± 11.1 BPM (which was 98.1 ± 4.6% of their calculated maximal HR) and a peak lactate of 9.2 ± 3.2 mmol/l at 557.1 ± 168.3 sec on our continuous speed, increasing slope treadmill protocol. Second division referees were younger, on a weekly average they trained more, achieved higher treadmill exercise time (respectively, 463.8 ± 131.9 vs 658.4 ± 143.9 sec, p < 0.001) and anaerobic threshold time (respectively, 265.8 ± 100.9 vs 348.2 ± 117.1 sec, p < 0.001), while the two different divisional referees had similar V̇O2max values. Regarding our physical fitness measurements, huge individual differences were observed between the referees (exercise time range: 259.0–939.0 sec, V̇O2max range: 25.3–62.4 ml/kg/min). Since it can affect their performance as referees, individual training planning, regular physical fitness measurements, and strict selection methods are suggested.