High-redshift quasars are important to study galaxy and active galactic nuclei (AGN) evolution, test cosmological models, and study supermassive black hole growth. Optical searches for high-redshift sources have been very successful, but radio searches are not hampered by dust obscuration and should be more effective at finding sources at even higher redshifts. Identifying high-redshift sources based on radio data is, however, not trivial. Here we report on new multi-frequency Giant Metrewave Radio Telescope (GMRT) observations of eight z > 4.5 sources previously studied at high angular resolution with very long baseline interferometry (VLBI). Combining these observations with those from the literature, we construct broad-band radio spectra of all 30 z > 4.5 sources that have been observed with VLBI. In the sample we found flat, steep and peaked spectra in approximately equal proportions. Despite several selection effects, we conclude that the z > 4.5 VLBI (and likely also non-VLBI) sources have diverse spectra and that only about a quarter of the sources in the sample have flat spectra. Previously, the majority of high-redshift radio sources were identified based on their ultra-steep spectra (USS). Recently a new method has been proposed to identify these objects based on their megahertz-peaked spectra (MPS). Neither method would have identified more than 18 per cent of the high-redshift sources in this sample. More effective methods are necessary to reliably identify complete samples of high-redshift sources based on radio data.