Floating offshore wind turbine (FOWT) systems are a fast-evolving technology, however, still have to gain economic competitiveness to allow commercial market uptake. Design optimization, focusing on cost reduction while ensuring optimum system performance, plays a key role in achieving these goals. Hence, in this work, an approach for optimizing a floating concept, utilizing global limit states, is developed. The optimization is carried out in Python, linked with Modelica and Dymola for modeling and simulation. For the FOWT design, the over-dimensioned OC3 spar-buoy is utilized. This is modified during the optimization regarding its geometrical dimensions and ballasting. The optimization criteria stability, mean and dynamic displacements, and tower top acceleration are used for formulating the objective functions. The optimization is carried out for one design load case, which is most critical for the considered criteria. Based on an initial study, NSGAII is chosen as optimizer. The convergence of the optimization is examined and the optimum design solution selected. In post-processing analyses, the overall performance of the optimized FOWT system is approved. The presented approach shows one example for the design optimization of a FOWT system and should deal as basis for more advanced design optimization tasks, including local characteristics and reliability aspects.