In this article, self-similarity and space-filling properties of fractal structures are explored as frequency selective diaphragms in rectangular waveguide, which can find applications in the design of compact, lightweight, and multiband waveguide filters with better out-of-band rejection ratio. Some self-affine fractal structures, based on Sierpinski gasket and plus shape fractals, are proposed, and the effect of scaling factor on the location of transmission bands is investigated. Self-similar structures like Hilbert curve, Koch curve, and Minkowski fractals are shown to be efficient in reducing resonant frequency of the diaphragm. Numerical results are presented along with the results obtained from HFSS, which show a good agreement. A typical application of the fractal diaphragm in the design of waveguide bandstop filter has also been demonstrated.