Highly infectious diseases, for instance, the recent novel corona virus pandemic (COVID-19) have impacted millions of lives in the globe. From its early stage, it's clear that its transmission is closely related to the crowd mobility. However, research on the predictive power of city level mobility is lacking. To fill this gap, in this paper, we propose the Inner-Inter city Learner (IIL), a method based on the high correlation between human interaction and the new cases, to conduct short and midterm predictions of the COVID-19 cases at a city level. In term of design, IIL is composed of two main components: an inter city transmission learner and an inner city propagation pattern learner. In the first module, we specially generate graphs based on the mobility between cities. Using graph neural networks fed with the city level mobility graphs, the model learns how the transmission pattern in a city is impacted by its neighborhood cities. And in the second part, capitalizing in the highly contagious nature of the virus, we use the human interactions within the cities to capture the invariant features that directly correlate with the spread. In addition, to account for the limited inner mobility data, we employed the model agnostic meta learning to transfer the features common to all cities. We conduct various experiments and compare our methods with the state-of-art baselines. The results show the superiority of our method in various forecast horizons.