We study, both analytically and numerically, an ARCH-like, multiscale model of volatility, which assumes that the volatility is governed by the observed past price changes over different time scales. With a power-law distribution of time horizons, we obtain a model that captures most stylized facts of financial time series: Student-like distribution of returns with a power-law tail, long-memory of the volatility, slow convergence of the distribution of returns towards the Gaussian distribution, multifractality and anomalous volatility relaxation after shocks. At variance with recent multifractal models that are strictly time reversal invariant, the model also reproduces the time asymmetry of financial time series: past large scale volatility influence future small scale volatility. In order to quantitatively reproduce all empirical observations, the parameters must be chosen such that the model is close to an instability, meaning that (a) the feedback effect is important and substantially increases the volatility, and (b) that the model is intrinsically difficult to calibrate because of the very long range nature of the correlations. By imposing consistency of the model predictions with a large set of different empirical observations, a reasonable range of the parameters value can be determined. The model can easily be generalized to account for jumps, skewness and multiasset correlations.