This paper is concerned with the design of a context-based fuzzy C-means (CFCM)-based multi-granular fuzzy model (MGFM) with hierarchical tree structures. For this purpose, we propose three types of hierarchical tree structures (incremental, aggregated, and cascaded types) in the design of MGFM. In general, the conventional fuzzy inference system (FIS) has problems, such as time consumption and an exponential increase in the number of if–then rules when processing large-scale multivariate data. Meanwhile, the existing granular fuzzy model (GFM) reduces the number of rules that increase exponentially. However, the GFM not only has overlapping rules as the cluster centers become closer but also has problems that are difficult to interpret due to many input variables. To solve these problems, the CFCM-based MGFM can be designed as a smaller tree of interconnected GFMs. Here, the inputs of the high-level GFMs are taken from the output to the low-level GFMs. The hierarchical tree structure is more computationally efficient and easier to understand than a single GFM. Furthermore, since the output of the CFCM-based MGFM is a triangular fuzzy number, it is evaluated based on a performance measurement method suitable for the GFM. The prediction performance is analyzed from the automobile fuel consumption and Boston housing database to present the validity of the proposed approach. The experimental results demonstrate that the proposed CFCM-based MGFM based on the hierarchical tree structure creates a small number of meaningful rules and solves prediction-related problems by making them explainable.