An adaptive reduced basis finite element heterogeneous multiscale method (RB-FE-HMM) is proposed for elliptic problems with multiple scales. The multiscale method is based on the RB-FE-HMM introduced in [A. Abdulle and Y. Bai, J. Comput. Phys, 2012, in press]. It couples a macroscopic solver with effective data recovered from the solution of micro problems solved on sampling domains. Unlike classical numerical homogenization methods, the micro problems are computed in a finite dimensional space spanned by a small number of accurately computed representative micro solutions (the reduced basis) obtained by a greedy algorithm in an offline stage. In this paper we present a residual-based a posteriori error analysis in the energy norm as well as an a posteriori error analysis in quantities of interest. For both type of adaptive strategies, rigorous a posteriori error estimates are derived and corresponding error estimators are proposed. In contrast to the adaptive finite element heterogeneous multiscale method (FE-HMM), there is no need to adapt the micro mesh simultaneously to the macroscopic mesh refinement. Up to an offline preliminary stage, the RB-FE-HMM has the same computational complexity as a standard adaptive FEM for the effective problem. Two and three dimensional numerical experiments confirm the efficiency of the RB-FE-HMM and illustrate the improvements compared to the adaptive FE-HMM.