We provide a procedure to identify the number of latent factors of stochastic volatility models. The methodology relies on the non-parametric Fourier estimation method introduced by Malliavin and Mancino (Finance Stoch 4:49–61, 2002) and applies to high-frequency data. Based on the Fourier analysis, we first estimate the latent volatility process and then the volatilities and covariances of the processes that are gradually identified, such as volatility of volatility and leverage. The analysis of the eigenvalue spectrum of the Gram matrix can reveal information about the actual number of factors driving the process at hand. We corroborate our analysis by numerical simulations on single and multi factor models. Finally, we apply our methodology to intraday prices from the S &P 500 index futures.