Varying index coefficient models (VICMs) proposed by Ma and Song (J Am Stat Assoc, 2014. ) are a new class of semiparametric models, which encompass most of the existing semiparametric models. So far, only the profile least squares method and local linear fitting were developed for the VICM, which are very sensitive to the outliers and will lose efficiency for the heavy tailed error distributions. In this paper, we propose an efficient and robust estimation procedure for the VICM based on modal regression which depends on a bandwidth. We establish the consistency and asymptotic normality of proposed estimators for index coefficients by utilizing profile spline modal regression method. The oracle property of estimators for the nonparametric functions is also established by utilizing a two-step spline backfitted local linear modal regression approach. In addition, we discuss the bandwidth selection for achieving better robustness and efficiency and propose a modified expectation-maximization-type algorithm for the proposed estimation procedure. Finally, simulation studies and a real data analysis are carried out to assess the finite sample performance of the proposed method.