This paper considers the problem of finding the position of a passive target using noisy time difference of arrival (TDOA) measurements, obtained from multiple transmitters and a single receiver. The maximum likelihood (ML) estimator’s objective function is extremely nonlinear and non-convex, making it impossible to use traditional optimization techniques. In this regard, this paper proposes the chaos-enhanced adaptive hybrid butterfly particle swarm optimization algorithm, named CAHBPSO, as the hybridization of butterfly optimization (BOA) and particle swarm optimization (PSO) algorithms, to estimate passive target position. In the proposed algorithm, an adaptive strategy is employed to update the sensory fragrance of BOA algorithm, and chaos theory is incorporated into the inertia weight of PSO algorithm. Furthermore, an adaptive switch probability is employed to combine global and local search phases of BOA with the PSO algorithm. Additionally, the semidefinite programming is employed to convert the considered problem into a convex one. The statistical comparison on CEC2014 benchmark problems shows that the proposed algorithm provides a better performance compared to well-known algorithms. The CAHBPSO method surpasses the BOA, PSO and semidefinite programming (SDP) algorithms for a broad spectrum of noise, according to simulation findings, and achieves the Cramer–Rao lower bound (CRLB).