In this paper the leakage effects in open laterally‐shielded multilayered microstrip transmission lines are investigated. Potential solutions are often reported to be difficult to find, since they are located in the complex propagating factor plane. To overcome this problem, a novel iterative algorithm is implemented to relate the solutions in the real axis of closed transmission lines with their corresponding complex open structure modes. The problem of modeling the leaky behavior of the open top wall is introduced, and a mathematical and physical explanation of the solutions is proposed. The advantage of this new technique is that it allows us to easily track the leaky solutions to their final location in the complex plane. Results are presented, including the field and power patterns associated to leaky waves, showing that the derived technique is indeed effective for the study of the complex modes excited in this type of structures. © 2003 Wiley Periodicals, Inc. Microwave Opt Technol Lett 37: 88–93, 2003; Published online in Wiley InterScience (www.interscience.wiley.com). DOI 10.1002/mop.10832