Mobile robots such as unmanned ground vehicles (UGVs) and unmanned aerial vehicles (UAVs) are increasingly used for precision agriculture. While UGVs have larger payload capabilities and longer operation time, they are limited to 2-D space. This makes UAVs better suited for tasks that require fast coverage, harsh terrain traversal, and high altitude or multilevel operation. However, it remains a challenging task to develop a reliable yet fully autonomous UAV system that can actively extract actionable information in large-scale cluttered agricultural environments. Such a system will have to estimate its own poses, build a map of the environment, navigate through obstacles, and act to gather information with limited onboard computation and battery life. In this survey, we first review recent advances in UAV hardware and software, ranging from novel platforms and sensors to state-of-the-art autonomous navigation, object detection and segmentation, robot localization, and mapping algorithms related to agriculture. We then provide a list of challenges in each field and potential opportunities for the broader adoption of UAVs in precision agriculture.