In order to enhance the braking safety and improve the fuel economy and emission of heavy-duty engines, a variable mode valve actuation system which can switch operation modes flexibly among four-stroke driving, two-stroke compression release braking, and cylinder deactivation modes on a four-stroke engine was developed in this article. The switching was controlled by the four-stroke driving module and the two-stroke braking module, both of which have two states: effective state and failure state. Besides, a full-cycle numerical model of a six-cylinder turbocharged engine was established for the performance analysis of two-stroke compression release braking mode. The orthogonal design method was introduced in the present study to obtain the optimum valve parameters which can result in high braking power and maintain the maximum cylinder pressure at a lower level at the same time. Then, the two-stroke compression release brake braking power with the optimum valve parameters was compared with four-stroke compression release braking power. Meanwhile, the two-stroke braking cam profile of the variable mode valve actuation system was designed according to the optimum valve parameters, and the two-stroke braking performance with the dynamic valve lift of the variable mode valve actuation system was validated by the numerical model including the hydraulic system. The results indicated that a higher engine speed leads to higher braking power at the same valve lift. Besides, two-stroke compression release brake braking power of the variable mode valve actuation system achieved 525.3 kW at 2600 r/min and 358.1 kW at 1900 r/min, 52.9% and 71.3% higher than four-stroke compression release braking power, respectively. Although the two-stroke compression release brake braking power with dynamic valve lift is slightly less than that with the optimum valve parameters, it is still much higher than that of the four-stroke compression release braking power. Therefore, it has a good application prospect for heavy-duty engines.