Soil organic matter (SOM) is an important soil property for agricultural production. Rising grain demand has increased the intensity of cultivated land development in the Sanjiang Plain of China, and there is a strong demand for SOM monitoring in this region. Therefore, Baoqing County of the Sanjiang Plain, an important grain production area, was considered the study area. In the study, we proposed a framework for high-accuracy SOM retrieval by coupling multi-temporal remote sensing (RS) images and variable selection algorithms. A total of 73 surface soil samples (0–20 cm) were collected in 2010, and Landsat 5 images acquired during the bare soil period (April, May, and June) were selected from 2008 to 2011. Three variable selection algorithms, namely, Genetic Algorithm, Random Frog and Competitive Adaptive Reweighted Sampling (CARS), were combined with Partial Least Squares Regression (PLSR) to build SOM retrieval models on the spectral bands and indices of the images. The results using a single-date image showed that the combination of variable selection algorithms and PLSR outperformed using PLSR alone, and CARS showed the best performance (R2 = 0.34, RMSE = 15.66 g/kg) among all the algorithms. Therefore, only CARS was applied to SOM retrieval in the different year interval groups. To investigate the effect of the image acquisition time, all images were divided into various year interval groups, and the resulting images were then stacked. The results using multi-temporal images showed that the SOM retrieval accuracy improved as the year interval lengthened. The optimal result (R2 = 0.59, RMSE = 11.81 g/kg) was obtained from the 2008–2011 group, wherein the difference indices derived from the images of 2009, 2010, and 2011 dominated the selected spectral variables. Moreover, the spatial prediction of SOM based on the optimal model was consistent with the distribution of SOM. Our study suggested that the proposed framework that couples stacked multi-temporal RS images with variable selection algorithms has potential for SOM retrieval.