The accurate forecasting of photovoltaic (PV) power generation is critical for smart grids and the renewable energy market. In this paper, we propose a novel short-term PV forecasting technique called the space-time convolutional neural network (STCNN) that exploits the location information of multiple PV sites and historical PV generation data. The proposed structure is simple but effective for multi-site PV forecasting. In doing this, we propose a greedy adjoining algorithm to preprocess PV data into a space-time matrix that captures spatio-temporal correlation, which is learned by a convolutional neural network. Extensive experiments with multi-site PV generation from three typical states in the US (California, New York, and Alabama) show that the proposed STCNN outperforms the conventional methods by up to 33% and achieves fairly accurate PV forecasting, e.g., 4.6–5.3% of the mean absolute percentage error for a 6 h forecasting horizon. We also investigate the effect of PV sites aggregation for virtual power plants where errors from some sites can be compensated by other sites. The proposed STCNN shows substantial error reduction by up to 40% when multiple PV sites are aggregated.