We study analytically and numerically the possibility of vortex modes propagation over planar dielectric rectangular waveguides, and consider the problem of waveguide geometry optimization for the support of vortex modes. The results show, that theoretically rectangular waveguides can provide transmission of quasi-TE and quasi-TM modes with high purity states of orbital angular momentum (OAM) in the dominant field component. However, only for the quasi-generate mode of azimuthal order ±1 the constituent eigenmodes can propagate in a phase-matched regime, and the vortex modes of higher azimuthal orders can propagate only with a certain beat length. Moreover, as the target azimuthal order increases, the normalized power of the corresponding OAM state in the modal superposition decreases. The analytical predictions have been verified by numerical electromagnetic simulations of silicon nitride waveguides providing field distributions and OAM spectra of the corresponding modal superpositions.