Coastal wetland soil organic carbon (CW-SOC) is crucial for both “blue carbon” and carbon sequestration. It is of great significance to understand the content of soil organic carbon (SOC) in soil resource management. A total of 133 soil samples were evaluated using an indoor spectral curve and were categorized into silty soil and sandy soil. The prediction model of CW-SOC was established using optimized support vector machine regression (OSVR) and optimized random forest regression (ORFR). The Leave-One-Out Cross-Validation (LOO-CV) method was used to verify the model, and the performance of the two prediction models, as well as the models’ stability and uncertainty, was examined. The results show that (1) The SOC content of different coastal wetlands is significantly different, and the SOC content of silty soils is about 1.8 times that of sandy soils. Moreover, the characteristic wavelengths associated with SOC in silty soils are mainly concentrated in the spectral range of 500–1000 nm and 1900–2400 nm, while the spectral range of sandy soils is concentrated in the spectral range of 600–1400 nm and 1700–2400 nm. (2) The organic carbon prediction model of silty soil based on the OSVR method under the first-order differential of reflectance (R′) is the best, with the Adjusted-R2 value as high as 0.78, the RPD value is much greater than 2.0 and 5.07, and the RMSE value as low as 0.07. (3) The performance of the OSVR model is about 15~30% higher than that of the support vector machine regression (SVR) model, and the performance of the ORFR model is about 3~5% higher than that of the random forest regression (RFR) model. OSVR and ORFR are better methods of accurately predicting the CW-SOC content and provide data support for the carbon cycle, soil conservation, plant growth, and environmental protection of coastal wetlands.