Abstract-Integral equation formulation and magnetic potentialGreen's dyadics for multilayered rectangular waveguide are presented for modeling interacting printed antenna arrays used in waveguidebased spatial power combiners. Dyadic Green's functions are obtained as a partial eigenfunction expansion in the form of a double series over the complete system of eigenfunctions of transverse Laplacian operator. In this expansion, one-dimensional characteristic Green's functions along a multilayered waveguide are derived in closed form as the solution of a Sturm-Liouville boundary value problem with appropriate boundary and continuity conditions. A method introduced here is based on the transmission matrix approach, wherein the amplitude coefficients of forward and backward traveling waves in the scattered Green's function in different dielectric layers are obtained as a product of transmission matrices of corresponding layers. Convergence of Green's function components in the source region is illustrated for a specific example of a two-layered, terminated rectangular waveguide.