We provide explicit small-time formulae for the at-the-money implied volatility, skew and curvature in a large class of models, including rough volatility models and their multi-factor versions. Our general setup encompasses both European options on a stock and VIX options, thereby providing new insights on their joint calibration. The tools used are essentially based on Malliavin calculus for Gaussian processes. We develop a detailed theoretical and numerical analysis of the two-factor rough Bergomi model and provide insights on the interplay between the different parameters for joint SPX-VIX smile calibration.