This paper studies the optimal three-dimensional (3-D) antenna array for direction finding when the antenna array is subject to geometric constraint. Two-dimensional (2-D) planar array has been widely used in existing direction finding systems, and the optimal geometry for planar array has also been investigated in existing works. However, the optimal antenna array in the 3-D space, especially when the positions of the antennas are subject to geometric constraint, has not been studied yet. In this paper, a novel approach based on coordinate transformation is proposed for optimal antenna array design in the 3-D space with geometric constraint. Due to geometric constraint, the position of each antenna has only two degreesof-freedom even thought it has three coordinate values, so that coordinate transformation can be used to convert 3-D coordinates of the array elements to 2-D ones. Then, existing optimization approach for 2-D planar array can be applied to address the issue of optimal 3-D antenna array. Simulation results show that the proposed antenna array in this paper can outperform the defacto uniform circular array (UCA) even in the presence of geometric constraint. As a potential application, the result in this paper can be applied to design the high-frequency (HF) direction finding antenna arrays that are placed in mountain area. INDEX TERMS Direction finding, optimal antenna array, three-dimensional antenna array, geometric constraint.