An analysis of quasi-optical unstable Bessel-Gauss resonator (QOUBGR) at millimeter wavelengths is presented in this paper. The QOUBGR, formed by a conical mirror and a convex mirror, is designed on the basic of quasi-optical theory and techniques. For the purpose of precisely analyzing the designed QOUBGR, a new algorithm known as iterative dyadic Green's functions (IDGF) is proposed, which originates from famous Fox-Li algorithm. The IDGF algorithm can calculate not only two-dimension (2-D) but also three-dimension (3-D) resonating modes in the cavity. Simulation results demonstrate that the designed QOUBGR can steadily support both zero-order and high-order resonant modes that are approximations to Bessel-Gauss beams. These beams will find their promising applications in the MM-and/or quasi-optical imaging and measurement systems.