Bridges and tunnels are large and complex structures that demand periodic inspections to assess their physical conditions. Although both have different designs and constructions from each other, a common problem they share is the drawbacks that their conventional inspections face. Moreover, conventional procedures not only are laborious, time-consuming, and costly, but also involve high and/or hard-to-reach places, often exposing the specialized inspectors to danger. To overcome these problems, the Unmanned Aerial Vehicle (UAV) is being explored to automate these inspections. Recently, the number of researches employing it within the civil infrastructure condition assessment has been growing in recent years, especially for the inspection of large and complex structures. Unlike the UAV-based bridge inspection that already has some review articles available in the literature, there are none yet for the tunnel inspection, to the best of authors' knowledge. Therefore, this article intends to conduct not only a review of the few UAV-based tunnel inspection researches available in the literature, but also an up-to-date review of UAV-based bridge inspection researches. Finally, the key challenges and future trends of the UAV-based inspection of these two structures are discussed, followed by the review conclusions.