Solar photovoltaics (PV) has advanced at an unprecedented rate and the global cumulative installed PV capacity is growing exponentially. However, the ability to forecast PV power remains a key technical challenge due to the variability and uncertainty of solar irradiance resulting from the changes of clouds. Ground-based remote sensing with high temporal and spatial resolution may have potential for solar irradiation forecasting, especially under cloudy conditions. To this end, we established two ultra-short-term forecasting models of global horizonal irradiance (GHI) using Ternary Linear Regression (TLR) and Back Propagation Neural Network (BPN), respectively, based on the observation of a ground-based sky imager (TSI-880, Total Sky Imager) and a radiometer at a PV plant in Dunhuang, China. Sky images taken every 1 min (minute) were processed to determine the distribution of clouds with different optical depths (thick, thin) for generating a two-dimensional cloud map. To obtain the forecasted cloud map, the Particle Image Velocity (PIV) method was applied to the two consecutive images and the cloud map was advected to the future. Further, different types of cloud fraction combined with clear sky index derived from the GHI of clear sky conditions were used as the inputs of the two forecasting models. Limited validation on 4 partly cloudy days showed that the average relative root mean square error (rRMSE) of the 4 days ranged from 5% to 36% based on the TLR model and ranged from 12% to 32% based on the BPN model. The forecasting performance of the BPN model was better than the TLR model and the forecasting errors increased with the increase in lead time.