The objective of this study is to simulate the performance of helical tube shell and tube heat exchanger with several optimization techniques using computational fluid dynamics CFD. To check the performance of a designed model of heat exchanger various techniques are available. In this study, the various possible model of the heat exchanger to enhance the performance of the device have been designed. Firstly, the straight tube is replaced by helical tube in the heat exchanger and used 10, 12, 14 number of helical baffles with 50% baffle cut. Total ten models have been developed. These models are model-I 4-turns without baffle, model-II 4-turns with 10 number baffles, model-III 5-turns without baffle, model-IV 5-turns with 12 number baffles, model-V 6-turns without baffle, model-VI 6-turns with 10 number baffles 0.083m baffle space, model-VII 6-turns with 12 number 0.083m baffle space, model-VIII 6-turns with 14 number baffles 0.064m baffle space, model-IX 7-turns without baffle, model-X 7-turns with 14 number baffles, different number of baffles and baffle space with 50% baffle cut and used CUO nanofluid model-XI 6-turns with 14 number baffle CUO fluid 0.083m baffle space CFD analysis simulation done on ANSYS FLUENT 18. The simulated result shows that the model XI is approximately 40% more optimized as compared to model-I and approximately 24% than model-VIII. It also found that the high heat transfer obtains with increased number of baffles.