Utilizing recycled asphalt pavements (RAP) in pavement construction is known as a sustainable approach with significant economic and environmental benefits. While studying the effect of high RAP contents on the performance of hot mix asphalt (HMA) mixes has been the focus of several research projects, limited work has been done on studying the effect of RAP fraction and particle size on the overall performance of high RAP mixes produced solely with either coarse or fine RAP particles. To this end, three mixes including a conventional control mix with no RAP, a fine RAP mix (FRM) made with 35% percent fine RAP, and a coarse RAP mix (CRM) prepared with 54% of coarse RAP were designed and investigated in this study. These mixes were evaluated with respect to their rutting resistance, fatigue cracking resistance, and low temperature cracking performance. The results indicate that although the CRM had a higher RAP content, it exhibited better or at least the same performance than the FRM. The thermal stress restrained specimen testing (TSRST) results showed that the control mix performed slightly better than the CRM, while the FRM performance was adversely affected with respect to the transition temperature midpoint and the maximum tensile stress temperature. Both of the RAP incorporated mixes exhibited better rutting resistance than the control mix. With regard to fatigue cracking, the CRM performed better than the FRM. It can be concluded that the RAP particle size has a considerable effect on its contribution to the total binder content, the aggregate skeleton of the mix, and ultimately the performance of the mix. In spite of the higher RAP content in the CRM versus FRM, the satisfactory performance observed for the CRM mix indicates a great potential in producing high RAP content mixes through optimizing the RAP particle size and content. The results also suggest that the black curve gradation assumption is not representative of the actual RAP particles contribution in a high RAP mix.