Soil organic carbon sequestration measures entail costs to farmers with different individual characteristics and located in different areas. A cost‐effective analysis taking into account these heterogeneities is crucial for developing effective public policy aimed at increasing carbon sequestration. We undertake such an analysis focusing on three soil organic carbon sequestration measures: no‐till, extension of temporary grasslands, and hedgerows. Through an optimization model applied to France, our results show that only extension of temporary grasslands can store carbon at low cost, though their potential for carbon sequestration is also low. For an ambitious carbon sequestration target, no‐till and hedgerows are needed. Because of heterogeneities among farmers, we show that economic incentives are more cost effective than command and control measures. Furthermore, we highlight that the carbon sequestration policy tested here is unable to totally neutralize agricultural greenhouse gas emissions and that a combination of agricultural and environmental policies is essential for tackling climate change.