In this work, the dynamic behavior of linear and nonlinear waves propagating at the separating surface between two thin layers of viscous Newtonian fluids is studied in the presence of the effect of insoluble surface surfactant. The two liquids are confined between two infinite rigid parallel plates and assumed to have different densities and viscosities. The equations of evolution for surface-wave elevation and concentration of surfactant are derived using the lubrication approximation. In the linear stage, by utilizing the normal mode approach, we have derived the dispersion relation that relates the wave angular frequency to the wave number and other parameters that is solved numerically to inspect the influences of some selected parameters on the stability criteria of the fluid flow. Also, analytical expressions for the growth rate as well as its maximum value with corresponding wave number are obtained in the special case of long-wave limiting. It is concluded that the Marangoni number $$\text {Ma}$$
Ma
has acquired a significant stabilizing influence on the fluid flow, whereas the inverse of the slippery length of substrate plate $$\beta$$
β
, resorts to the destabilize the motion of the interfacial waves. Consequently, both of the Marangoni number and the substrate slippy coefficient can be utilized to control the film flow regime, where they preserve the film laminar flow and tend to prevent the film breakdown. These can be useful in many industrial applications such as coating processes, heat exchangers, cooling microelectronic devices, chemical reactors, food processing, thermal protection design of combustion chambers in rocket engines and operation of Laser cutting and heavy casting production processes.