The perfectly matched layer (PML) is a widely used tool to truncate the infinite domain in modal analysis for optical waveguides. Since the PML mimics the unbounded domain, propagation modes and leaky modes of the original unbounded waveguide can be derived. However, the presence of PML will introduce a series of new modes, which depend on the parameters of PML, and they are named as Berenger modes. For twodimensional step-index waveguides, the eigenmode problem is usually transformed into an algebraic equation by the transfer matrix method (TMM). When the waveguide is nonhomogeneous, in which the refractive index in the core is varied, TMM is not available. In this paper, we use the differential TMM to derive the dispersion relation. We also deduced the asymptotic formulas for leaky modes and Berenger modes separately, which are accurate for large modes.