Low-frequency modes that develop as a result of an instability in a dusty rf discharge plasma were studied experimentally, leading to an empirical explanation for the instability. In the experiment, particle diameter grew with time. Two instability modes appeared after growth to a sufficient size. A filamentary mode appeared abruptly, and later a great void mode developed as a dust-free region with an intense glow inside and a sharp boundary outside. These modes were characterized by two-dimensional laser light scattering, video imaging, optical emission spectroscopy, Langmuir probe measurements, and Fourier analysis of the fluctuation spectrum. Dust growth was measured by electron microscopy and optical extinction, yielding the dust particle size and dust number density. The electron density was found to be enhanced inside the great void, due to an absence of electron depletion on the dust grains. The great void was explained by the ion drag force, which becomes stronger than the opposing Coulomb force once the particle size reaches a critical diameter. When a dust-free region develops, its electron density is enhanced, the ionization rate increases, and the ion flow that pushes particles outward is further augmented. The plasma used in the experiment grew particles by sputtering of the electrodes, although the same instabilities are expected to occur in other types of dusty plasma discharges as well. ͓S1063-651X͑99͒00801-6͔