Time–frequency analysis is one of the effective methods for seismic thin‐layer detection. Conventional time–frequency analysis technology for seismic thin‐layer detection is interfered by the energy of adjacent frequency signals, and there is information redundancy in the frequency‐domain analysis. Therefore, an S transform with improved window factor, which is based on the constrained non‐negative matrix factorization, is constructed to realize seismic thin‐layer detection. First, the seismic data is processed by the S transform of the improved window factor, and then we can obtain the frequency‐domain information with strong time–frequency focus by changing the adjustment factor and attenuation factor in the window function. Furthermore, the key frequency of the seismic data spectrum, which can also be called the key frequency characteristic factor, can be calculated by the non‐negative matrix factorization algorithm. Fortunately, the overthrust model shows a good correspondence between the key frequency characteristic factor and the thin‐layer interface. The field data example shows that this approach provides a new approach for thin‐layer detection.