Source localization is the process of assigning spatial metadata to a set of unlabeled time of arrival for acoustic signals that may be interlaced with extreme outliers caused by detection and material-induced errors. Exploiting the fact that location error is attributed to high residuals, a new framework for non-linear source localization is proposed in this article. The detected time of arrival was employed to calculate the intermediate source and event time. The framework then quarantines sensors with high residual based on the intermediate event time and relative distance from the intermediate source. Then, the time of arrivals for quarantined sensors were reviewed and adjusted. Finally, with new arrival time, the framework iteratively optimizes the spatial location and event time, along with the updating of velocity field through a feedback loop. The new framework was used to localize acoustic emission sources with extreme outliers in an anisotropic medium. The results indicate that the proposed framework can accurately quarantine the bad sensor and localize sources with an accuracy better than existing source localization methods. Finally, the framework was employed to study the progressive damage in a steel fiber–reinforced beam under four-point bending. The source localization result is consistent with that from visual inspection of the member at failure.