The effects of kinetic whistler wave instabilities on the runaway-electron (RE) avalanche is investigated. With parameters from experiments at the DIII-D National Fusion Facility, we show that RE scattering from excited whistler waves can explain several poorly understood experimental results. We find an enhancement of the RE avalanche for low density and high electric field, but for high density and low electric field the scattering can suppress the avalanche and raise the threshold electric field, bringing the present model much closer to observations. The excitation of kinetic instabilities and the scattering of resonant electrons are calculated self-consistently using a quasilinear model and local approximation. We also explain the observed fast growth of electron cyclotron emission signals and excitation of very low-frequency whistler modes observed in the quiescent RE experiments at DIII-D tokamak. Simulations using ITER parameters show that by controlling the background thermal plasma density and temperature, the plasma waves can also be excited spontaneously in tokamak disruptions and the avalanche generation of runaway electrons may be suppressed.