In order to meet the demand of increasing mobile data traffic and provide better user experience, heterogeneous cellular networks (HCNs) have become a promising solution to improve both the system capacity and coverage. However, due to dense self-deployment of small cells in a limited area, serious interference from nearby base stations may occur, which results in severe performance degradation. To mitigate downlink interference and utilize spectrum resources more efficiently, we present a novel graph-based resource allocation and interference management approach in this paper. Firstly, we divide small cells into cell-clusters considering their neighborhood relationships in the scenario. Then, we develop another graph clustering scheme to group user equipment (UE) in each cell-cluster into UE-clusters with minimum intra-cluster interference. Finally, we utilize a proportional fairness scheduling scheme to assign subchannels to each UE-cluster and allocate power using waterfilling method. In order to show the efficacy and effectiveness of our proposed approach, we propose a dual-based approach to search for optimal solutions as the baseline for comparisons. Furthermore, we compare the graph-based approach with the state of the arts and a distributed approach without interference coordination. The simulation results show that our graph-based approach reaches more than 90 percent of the optimal performance, and achieves a significant improvement in spectral efficiency compared to the state of the arts and the distributed approach both under cochannel and orthogonal deployments. Moreover, the proposed graph-based approach has a low computation complexity, making it feasible for real-time implementation.