Much interest has arisen in the problem of detecting weak optical subpixel objects in a sequence of images immersed in a heavy homogeneous Gaussian clutter background. In optical systems, the presence of the objects changes the background plus the channel noise covariance matrix. Hence, this matrix may be different under null and alternative hypotheses. Because the maximum likelihood estimate of the background covariance matrix is not available when using the primary data set, we use an approximate estimate of a certain number of eigenvalues of the background covariance matrix that are available for estimation. We derived the general likelihood ratio test for the problem at hand and obtained the modified spectrum matching detector (MSpMD). Using the simulation results, we contrast it with well-known spectrum matching detectors and the hypothesis-independent matched subspace detectors. We also present experimental results of small subpixel objects detection on an agitated sea surface, using digital optical experiments that demonstrate the validation of the simulating results. We show the advantage of the MSpMD that allows to detect small subpixel floating objects on an agitated sea surface even with the same means and variances of the reflected signals from the sea surface and the object. © The Authors. Published by SPIE under a Creative Commons Attribution 4.0 Unported License. Distribution or reproduction of this work in whole or in part requires full attribution of the original publication, including its DOI.