Braking safety of heavy-duty engines has always been the focus of the research, and the fuel economy and after-treatment thermal management during low-load operation of heavy-duty engines have also received much attention in recent years. A variable mode valve actuation system which can realize switching between four-stroke driving, two-stroke compression release braking and cylinder deactivation modes on a traditional four-stroke engine was proposed in this article. Two-stroke compression release braking mode of the variable mode valve actuation system can greatly enhance the braking safety, while the overload of valve train was a great challenge, especially during the release event. The effects of different release opening timing on cylinder pressure and the braking performance were studied. The results indicated that a higher cylinder pressure does not always lead to higher braking power. When the release opening timing was advanced by 6 °CA, the braking power reduced by only 9 kW (2.65%) at 1900 r/min compared with the initial value, while the maximum cylinder pressure reduced by 11.4 bar (20.8%). Besides, the variable mode valve actuation system can realize alternate three-cylinder cylinder deactivation mode on a six-cylinder turbocharged engine, which can improve the brake-specific fuel consumption by 14.67% and increase the turbine outlet temperature by 63.6 °C and reduce the exhaust flow rate by 50.66% at lightly load idle. Meanwhile, when the engine load is less than 50% at the rated speed, the three-cylinder cylinder deactivation mode can improve the brake-specific fuel consumption, increase the turbine outlet temperature and reduce the exhaust flow rate. The increase of the turbine outlet temperature and the decrease of the exhaust flow rate are very beneficial to improve the efficiency of the after-treatment thermal management of heavy-duty engines.