Classification of hyperspectral images usually suffers from high dimensionality and few reference data, which limits the performance of the pixelwise classifiers. The spectral-spatial classifiers, which integrate the spectral data and the spatial information during the classification, perform impressively in terms of the high classification accuracy and the homogeneous appearance of the classification map. In this paper, we propose a new probabilistic framework for spectral-spatial classification (PFSSC), which integrates the spectral data and the spatial information from the probabilistic point of view. Both the spectral data and the spatial information are used to estimate the per-pixel probability, which gives the likelihood that one pixel belongs to one class, respectively. The classification map can then be directly derived from the joint probability. In the proposed framework, a pixelwise probabilistic classifier can be extended as a spectral-spatial one since it can integrate spatial information easily. Furthermore, these spectral-spatial classifiers in the proposed framework are realized in an iterative way to avoid the problem caused by the limited reference data to some extent. In each iterative step, some unassigned pixels are classified by considering the pixels assigned in previous iterative steps. In this iterative process, pixels are assigned to specific labels step by step gradually. In the proposed framework, the probabilistic support vector machine (SVM) and random forest (RF) are extended to be two spectral-spatial classifiers. In short, we denote them as SVM-PFSSC and RF-PFSSC, respectively. The experimental results show that SVM-PFSSC and RF-PFSSC outperform some pixelwise and spectral-spatial classifiers.