The presence of a “modal noise” component leads to estimator instability when Capon's maximum likelihood (ML) method is applied to the processing of data from a vertical array in an acoustic waveguide. The physics of the waveguide forces signal vectors and noise vectors alike to be projected onto the span of the “mode” vectors, when the number of sensors (N) exceeds the number of propagating modes (M). The instability occurs whenever the (single snapshot) N × 1 data vectors have the form x = Us + Uγ + white noise, where the matrix U is N × M (sampling the normal modes at the hydrophone locations and independent of the actual acoustic disturbances present), and s and γ correspond to signal and ambient noise sources, respectively. This condition arises in normal-mode and local normal-mode propagation. The dominant eigenvectors of R−1 (where R is the cross-spectral matrix) are sensitive to slight inaccuracies in the calculation of R−1 in ways that affect the performance of the ML estimator. Following transformation of the N × N matrix R to the M × M modal space cross-spectral matrix T, Capon's method is applied to T to obtain the “reduced maximum likelihood” (RML) estimator. This procedure, which is a development of the sector focused stability technique of Steele and Byrne [Proceed. ISSPA 87, 24–28 August 1987, Brisbane, Australia, pp. 408–412], largely eliminates instabilities due to inaccurate inversion of R. Simulations are presented for a shallow-water environment to provide comparison between the ML and the RML estimators. These indicate that the degree of instability depends upon the level of noise (both correlated noise and white noise) and that a significant improvement in performance can be expected by use of the RML estimator in both cases.

Capon’s maximum likelihood (ML) method has been used with some success in matched field processing for source range and depth estimation. One reason for the interest in the ML is that it is data adaptive; that is, it adapts to the actual noise field present rather than requiring an a priori estimate of the noise component for prewhitening. When modal noise is present the ML can become sensitive to any deviations from the unperturbed case (i.e., from the model) as would be introduced by phase errors or model parameter errors. Using a dimensionality-reduction procedure a more stable data adaptive method, the ‘‘reduced’’ ML (RML), is obtained. The RML is compared here with the ML on simulated data from a 21-sensor array in a Pekeris waveguide supporting eight normal modes. Under modal noise conditions the RML provides a significant improvement over ML when phase errors occur. Although the deviation from the model considered here is that caused by phase errors, the nature of the perturbation is not important since the sensitivity of ML is not to any special type of perturbation.