The existing adaptive echo cancellation based howling (typically in hearing aids) removal methods have several drawbacks such as insufficient attenuation of the howling component, slow response and nonlinear distortion. To solve these problems, we propose a segmented notch filtering based scheme. Specifically, firstly, it is proved that the attenuation value can reach −330 dB at any detected howling frequency; secondly, the filter coefficients can be readily calculated by a closed-form formula, yielding a fast response to the sudden howling accident; thirdly, the closed-form formula of this filter is theoretically an even function, indicating that this filter possesses a linear transfer characteristic. In combination with proper segmentation and precisely removing these transient samples arising from FIR (Finite Impulsive Response) filtering, nonlinear distortion can be entirely avoided. Experimental results show that our proposed scheme can not only accurately estimate the howling frequency, but can also completely remove it, which yields a high-quality output waveform with a recovery SNR of about 22 dB. Therefore, the proposed segmented notching based scheme possesses vast potential for hearing aid development and other relevant applications.