An excellent mission plan is the prerequisite for the acquisition of high quality airborne hyperspectral images which are useful for environmental research, mining etc. In order to minimize the radiance non-uniformity caused by the anisotropic reflectance of targets, the flight direction is mostly designed on the solar azimuth or 180° from it for whiskbroom and pushbroom imagers. However, the radiance to the observer is determined not only by the reflectance of the target, but also by the terrain, the illumination direction and the observation direction. So, the flight direction which is defined to minimize radiance non-uniformity might change with the terrain. In order to find the best flight direction for rugged terrain, we firstly analyze the causes of the effect of terrain on radiation non-uniformity based on the radiative transfer process. Then, the flight direction design method is proposed for composite sloping terrain. Tested by digital and physical simulation experiments, the radiance non-uniformity is minimized when the aircraft flies in the designated direction. Finally, a workflow for flight direction planning and optimizing is summarized, considering the flight mission planning techniques and the workflow of remote sensing missions.