In this paper, a transcritical carbon dioxide power generation system based on a vortex tube is studied, which has the advantage of the self-condensation of carbon dioxide. The thermodynamic performance of the system was investigated by establishing a mathematical model. The results showed that under fundamental working conditions, the system could output a net power of 271.72 kW, and the thermal efficiency as well as the exergy efficiency of the system could reach 7.38% and 27.09%, respectively. Exergy analysis showed that the turbine had the greatest exergy loss among the system’s components, followed by the vortex tube, pump, heater and cooler. Parameter analysis showed that increasing the outlet pressure and inlet temperature of the vortex tube can improve the thermal efficiency and exergy efficiency of the system. In addition, the improvement in the turbine component’s efficiency is the most beneficial to the system’s performance, among which the turbine’s efficiency has the greatest impact. Carbon dioxide can be effectively liquified by expanding it in the vortex tube, and its liquefaction ratio increases with the decrease in the vortex tube’s inlet temperature and the increase in the vortex tube’s inlet pressure.