There have been extensive works on the design of opportunistic spectrum access and routing schemes to improve spectrum efficiency in Cognitive Radio Networks (CRNs), which becomes an integral component in the future communication regime. Nonetheless, the potentials of CRNs in boosting network performance yet remain to be explored to reach the full benefits of such a phenomenal technique. In this paper, we study the end-to-end latency in CRNs in order to find the sufficient and necessary conditions for real-time applications in finite networks and large-scale deployments. We first provide a general mobility framework which captures most characteristics of the existing mobility models and takes spatial heterogeneity into account. Under this general mobility framework, secondary users are mobile with an mobility radius α, which indicates how far a mobile node can reach in spatial domain. We find that there exists a cutoff point on α, below which the latency has a heavy tail and above which the tail of the latency is bounded by some Gamma distributions. As the network grows large, the latency is asymptotically scalable (linear) with respect to the dissemination distance (e.g., the number of hops or Euclidean distance). An interesting observation is that although the density of primary users adversely impacts the expected latency, it makes no influence on the dichotomy of the latency tail in finite networks and the linearity of latency in large networks. Our results encourage CRN deployment for real-time and large applications, when the mobility radius of secondary users is large enough.