Follicular lymphoma (FL) is a cancer of lymph system and it is the second most common lymphoid malignancy in the western world. Currently, the risk stratification of FL relies on histological grading method, where pathologists evaluate hematoxilin and eosin (H&E) stained tissue sections under a microscope as recommended by the World Health Organization. This manual method requires intensive labor in nature.Due to the sampling bias, it also suffers from inter-and intra-reader variability and poor reproducibility. We are developing a computer-assisted system to provide quantitative assessment of FL images for more consistent evaluation of FL. In this study, we proposed a statistical framework to classify FL images based on their histological grades. We introduced model-based intermediate representation (MBIR) of cytological components that enables higher level semantic description of tissue characteristics. Moreover, we introduced a novel color-texture analysis approach that combines the MBIR with low level texture features, which capture tissue characteristics at pixel level. Experimental results on real follicular lymphoma images demonstrate that the combined feature space improved the accuracy of the system significantly. The implemented system can identify the most aggressive FL (grade III) with 98.9% sensitivity and 98.7% specificity and the overall classification accuracy of the system is 85.5%.