As a typical smart material, giant electrorheological fluid (GERF) has a greater yield stress than electrorheological fluid (ERF) for vibration isolation. However, as rheological material, the modeling precision is severely degraded by its innate rate-dependent hysteretic nonlinearity and uncertainty. In this paper, a novel control method is proposed, which requires little information about the damper based on GERF. The proposed method combines nonlinear fractional-order control (FC) and extended state observer (ESO) by constructing damper as a second-order disturbance-based (SODB) structure to handle hysteretic nonlinearities, dynamic uncertainties and unknown disturbances. In contrast to the prevalent model-free control that neglects hysteresis nonlinearity, the proposed control algorithm considers it as a general disturbance and eliminates it. In addition, compared with linear FC with complex fractional-order ESO, where the order needs to be known in advance, nonlinear FC has improved robustness for uncertain fractional-order systems only via pure ESO. Simulations and experimental results demonstrate that the nonlinear controller outperforms the linear counterpart and the proposed method exhibits superior control performance compared to the existing model-free control, with an improvement of 26.9%.