Speeding up cyclic update schemes by pattern searches

Abstract

A popular strategy for dealing with large parameter estimation problems is to split the problem into manageable subproblems and solve them cyclically one by one until convergence. We address a well-known problem with this strategy, namely slow convergence under low noise. We propose using so called pattern searches which consist of a parameter-wise update phase followed by a line search. The search direction of the line search is computed by combining the individual updates of all subproblems. The approach can be used to accelerate learning of several methods proposed in the literature without the need for large algorithmic modifications such as evaluation of global gradients. The proposed modification is shown to reduce the convergence time in a realistic independent component analysis (ICA) problem by more than 85 %.

Nonlinear Source Separation

“…In [46,50] a faster optimization method has been proposed, which can be used both for ensemble learning and for certain other classes of problems. In [49] an improvement of the basic nonlinear separation method was proposed, which avoids some of the approximations of the basic method, and is both more reliable and more accurate.…”