For the exploitation of wind energy, planning/designing a wind farm plays a crucial role in the development of wind farm project, which must be implemented at an early stage, and has a vast influence on the stages of operation and control for wind farm development. As a step of the wind farm planning/designing, optimizing the wind turbine placements is an effective tool in increasing the power production of a wind farm leading to an increased financial return. In this paper, the optimization of an offshore wind farm with an irregular boundary is carried out to investigate the effectiveness of grid and coordinate wind farm design methods. In the study of the grid method, the effect of grid density on the layout optimization results is explored with 20 × 30 and 40 × 60 grid cells, and the means of coping with the irregular wind farm boundary using different wind farm design methods are developed in this paper. The results show that, depending on the number of installed wind turbines, a power output increase from 1% to 1.5% is achieved by increasing the grid density from 20 × 30 to 40 × 60. However, the computational time is more than doubled, rising from 23 h to 47 h with 40 wind turbines being optimized from the coarse grid cells to the densified grid cells. In comparison, the coordinate method is the best option for achieving the largest power increase of 1.5% to 2% (relative to the coarse 20 × 30 grid method), while the least computational time (21 h with 40 wind turbines optimized) is spent.