With the continuous exploitation of offshore natural gas, the content of CO2 produced gradually increases. It is not economical to separate more CO2 from natural gas after transportation, and more CO2 will aggravate the corrosion of pipelines. The commonly used decarburization process is not suitable for offshore platforms, and there are problems of high energy consumption and large space occupation. Therefore, dense phase separation of associated gas with high carbon dioxide content is a better separation method. In this paper, the equation of state is optimized by comparing the experimental and CO2 system phase characteristics simulation. Based on the selected equation of state (EOS), a three-level separation model of phase equilibrium characteristics is established. The separation efficiency is simulated to complete the separation of CO2 and methane. The separation process is optimized by a genetic algorithm, and the temperature and pressure under the best separation efficiency are determined. The PR-EOS was selected as the equation with the highest calculation accuracy. Through process simulation and algorithm optimization, the best separation efficiency was 72.23%.