The most commonly used subsea pipeline installation method is the S-Lay method. A very important and complex task in an S-Lay installation engineering analysis is to find the optimal pipelay vessel installation configuration for every distinctive pipeline route section. Installation loads in the pipeline are very sensitive to small changes in the configuration of the pipeline supports during laying and other influential parameters, such as the tensioner force, stinger angle, trim and draft of the pipelay vessel. Therefore, the process of an engineering installation analysis is very demanding, and there is a need for an automated optimization process. For that purpose, installation engineering methodology criteria and requirements are formalized into a nonlinear optimization problem with mixed continuous and discrete variables. A special tailored multi-objective genetic algorithm is developed that can be adjusted to any desired combination of criteria and offshore standards’ requirements. The optimization algorithm is applied to the representative test cases. The optimization procedure efficiency and quality of the achieved solution prove that the developed genetic algorithm operators and the whole optimization approach are adequate for the presented application.