To reduce the negative effect on sound source localization when the source is at an extreme angle and improve localization precision and stability, a theoretical model of a three-plane five-element microphone array is established, using time-delay values to judge the sound source’s quadrant position. Corresponding judgment criteria were proposed, solving the problem in which a single-plane array easily blurs the measured position. Based on sound source geometric localization, a formula for the sound source azimuth calculation of a single-plane five-element microphone array was derived. The sinusoids and cosines of two elevation angles based on two single-plane arrays were introduced into the sound source spherical coordinates as composite weighted coefficients, and a sound source localization fusion algorithm based on a three-plane five-element microphone array was proposed. The relationship between the time-delay estimation error, elevation angle, horizontal angle, and microphone array localization performance was discussed, and the precision and stability of ranging and direction finding were analyzed. The results show that the measurement precision of the distance from the sound source to the array center and the horizontal angle are improved one to threefold, and the measurement precision of the elevation angle is improved one to twofold. Although there is a small error, the overall performance of the sound source localization is stable, reflecting the advantages of the fusion algorithm.