Carrier Aggregation (CA) is one of the most important technologies in the LTE-Advanced (LTE-A) system for better transmission efficiency. The basic idea of CA is to adopt more than one component carrier (CC) for user equipment (UE) to increase its bandwidth. However, when UE adopts component carriers from other adjacent base stations, it will cause the interference and may decrease transmission efficiency especially for the cell edge UE. Therefore, solving interference with carrier aggregation is a critical issue toward the effective improvement of system throughput. In this paper, we propose a novel method to deal with not only the interference between pico station and macro base station, but also the interference between base stations. In the proposed scheme, the radio interference, channel condition of UE, and load balance are taken into consideration for proper CC selection so as to improve the cell-edge UE performance. The proposed scheme considers the interaction and correlation between Inter-cell Interference Coordination (ICIC) and enhanced Inter-cell Interference Coordination (eICIC) to arrange the secondary CC for LTE-A UE. The simulation results show that the proposed scheme can increase transmission efficiency especially for the edge UE.