The present study focuses on the Rayleigh equation derived in the case of a Hall thruster plasma having dust contamination, produced near the exit of the channel due to the finite divergence of the accelerated ions and the sputtering of the walls. In the presence of negatively charged dust particles, a modified form of the well-known Rayleigh equation of fluid dynamics is realized. The modified equation takes the form of the Rayleigh equation for a particular band of oscillation frequency, revealing that the Rayleigh instability shall occur in the thruster only for the oscillations having frequencies within this band. For better understanding, the variation of frequency band with various parameters, viz. dust density, dust charge number, dust mass, electron temperature and ion density, has been traced out.