Modal computation with high precision is an important issue for wave propagation in optical waveguides when the eigenmode expansion method is applied. This numerical study of an open waveguide is accomplished by terminating the unbounded regions with perfectly matched layers. Then, by the differential transfer matrix method (DTMM), two nonlinear relations of modes for the TM case are proposed for complex waveguides with varied refractive indexes along the transverse direction. For the gradually varied waveguides, a higher approximation is provided and some asymptotic formulas of leaky modes are created, which may be used as better initial values for some iteration methods. Numerical examples illustrate that our treatment is highly feasible and effective.