Cognitive radio networks significantly improve spectrum utilization efficiency by allowing unlicensed users to opportunistically access the unused licensed spectrum. One of the key challenges is to determine how to properly assign the unused spectrum slots without impacting the licensed users and if necessary, move the existing users to different spectrum bands. In this paper, we present a spectrum migration model and propose a simulated annealing approach to determine the proper spectrum migration solution, which optimizes the aggregated utility while satisfying capacity constraints, interference constraints, and rate requirement constraints. The algorithm also takes into account different classes of traffic. We show that the algorithm greatly outperforms the greedy migration approach and other assignment approaches without migration.