Thermoacoustic instabilities are characterized by self-sustained largeamplitude pressure oscillations, which are produced by the interaction between unsteady heat release and acoustic waves. Such instabilities are detrimental to land-based gas turbine and aero-engines. To mitigate thermoacoustic instabilities, the coupling between unsteady heat release and pressure perturbations must somehow be interrupted. Feedback control techniques with a dynamic controller implemented can be used to achieve this. One of the most classical controllers is based on a finite-or infinite-impulse response filter, whose coefficients are optimized using least mean square algorithm (LMS). In this work we experimentally investigated and compared the performance of four LMS-based algorithms on stabilizing an unstable thermoacoustic system. Effects of the step size and filter length are studied one at a time. It is found that the filter length plays an important role in determining the convergence speed and steady error. In addition, the step size involved in the LMS algorithms is shown to be varied over a wide range. However, when the step size is small, simplified/revised LMS-based algorithms perform better than the conventional one in the absence of a feedback term. However, with the step size increased, these algorithms behave similarly. In order to improve the performance of conventional LMS-based algorithms, a time-varying step-size controller is developed and experimentally implemented. Approximately 50 dB sound reduction is achieved. The controller is also demonstrated to be able to track and prevent the onset of new limit cycle resulting from the changes of fuel flow rate. Finally, the transfer function of the thermoacoustic system is measured by injecting broad-band white noise.