We study the effect of filter anisotropy and sub-filter scale (SFS) dynamics on the accuracy of large eddy simulation (LES) of turbulence, by using several types of SFS models including the dynamic Smagorinsky model (DSM), dynamic mixed model (DMM), and the direct deconvolution model (DDM) with the anisotropic filter. The aspect ratios (AR) of the filters for LES range from 1 to 16. We show that the DDM is capable of predicting SFS stresses accurately at highly anisotropic filter. In the a priori study, the correlation coefficients of SFS stress reconstructed by the DDM are over 90%, which are much larger than those of the DSM and DMM models. The correlation coefficients decrease as the AR increases. In the a posteriori studies, the DDM outperforms DSM and DMM models in the prediction of various turbulence statistics, including the velocity spectra, and probability density functions of the vorticity, SFS energy flux, velocity increments, strain-rate tensors and SFS stress. As the anisotropy increases, the results of DSM and DMM become worse, but DDM can give satisfactory results for all the filter-anisotropy cases. These results indicate that the DDM framework is a promising tool in developing advanced SFS models in the LES of turbulence in the presence of anisotropic filter.