This paper shows development challenges for 11-liter heavy-duty off-highway diesel engines to meet Tier 3 emission regulations with a base diesel engine compliant with Tier 2 emission regulations. In the case of the installation of an exhaust gas recirculation (EGR) system for reduction of NOx emissions, there exists a risk of increased particulate matters (PM) emissions. An in-cylinder PM reduction is still necessary since a diesel particulate filter (DPF) after-treatment system is not under the consideration. The objective of this research is to see whether the base engine has a potential to meet Tier 3 emission regulations by changing in-cylinder configuration parameters including the bowl shape, injector position, the number of intake and exhaust valves, injector tip protrusion, and injector tip specifications such as nozzle spray angle and nozzle flow rate. These parameters are very important parts which enhance the air and fuel mixing process that helps the combustion process. Thus, the optimization of these design variables is essential to improve combustion efficiency and emissions reduction. In this study, the multi-dimensional computational fluid dynamics (CFD) code KIVA-3V is used to perform combustion simulations. The Kelvin–Helmholtz/Rayleigh–Taylor (KH-RT) model is employed for spray breakup and a reduced chemical mechanism for n-heptane is employed to simulate ignition delay and combustion of diesel fuel. To verify the simulation results, engine bench tests were performed with installations of the final version of in-cylinder geometry in C1-8 mode which is one of main test cycles to meet Tier 3 emission regulations. Finally, Tier 3 emission regulations have been met with the currently optimized in-cylinder configuration parameters.