This paper presents the design and implementation of a vision-based automatic guidance system on a fixed-wing unmanned aerial vehicle (UAV). The system utilises a low-cost ordinary video camera and simple but efficient image processing techniques widely used in computer-vision technology. The paper focuses on the identification and extraction of geographical tracks such as rivers, coastlines, and roads from real-time aerial images. The image processing algorithm primarily uses colour properties to isolate the geographical track of interest from its background. Hough transform is eventually used to curve-fit the profile of the track which yields a reference line on the image plane. A guidance algorithm is then derived based on this information. In order to test the vision-based automatic guidance system in the laboratory without actually flying the UAV, a hardware-in-the-loop simulation system is developed. Description regarding the system and significant simulation result are presented in the paper. Finally, an actual test flight where the UAV successfully follows a stretch of a river under automatic vision-based guidance is also presented and discussed.