Shallow lake sediments have huge potential for carbon storage, but they are vulnerable to various environmental changes and are highly susceptible to becoming a source of carbon emissions. Understanding the amount of sediment carbon storage can provide information about the potential of shallow lakes in mitigating climate change. In this study, sediment sampling was conducted at 12 study sites in Chaohu Lake, China, and sediment water content, grain size, bulk density, and sediments organic carbon (SOC) content were examined in five layers along the vertical direction, respectively, and the distribution and storage of SOC were estimated. The results showed that the surface sediments in the west lake area of Chaohu Lake would release 66,300 tons of SOC to the overlying water body in the future. The sediments in the middle lake area and the east lake area will absorb another 15,900 t and 17,300 t of TOC from the water body in the future, respectively. Overall, the lake-wide sediments will release 33,100 t of SOC into the water body in the future. In addition, the results of the study also indicate that human activities are another major influence on the change in organic carbon stocks in lake sediments, and therefore, proactive measures for the restoration and protection of lake sediments are essential because increasing the SOC stocks in the sediments and maintaining the lakes in a sustainable manner can contribute to the crucial role they play in mitigating climate change.