The capabilities of autonomous mobile robotic systems have been steadily improving due to recent advancements in computer science, engineering, and related disciplines such as cognitive science. In controlled environments, autonomous robots have been able to achieve relatively high levels of autonomy. In more unstructured environments, however, the realisation of autonomous mobile robots remains challenging due to limitations in the robots’ external environment understanding. Many autonomous mobile robots use classical, learning-based or hybrid approaches for navigation. The classical navigation approach typically includes robot perception, localisation, environmental mapping, path planning and motion control stages. More recent learning-based methods may replace the complete navigation pipeline or selected stages of the classical approach. For effective deployment, autonomous robots need to be able to understand their external environments at a sophisticated level according to their intended applications. Therefore, in addition to robot perception, scene analysis and higher-level scene understanding (e.g., traversable/non-traversable, and rough or smooth terrain) are required for autonomous robot navigation in unstructured outdoor environments. A wide number of alternative approaches have been proposed in recent years to attempt to address these scene understanding requirements. This paper provides a comprehensive review and critical analysis of these methods in the context of their applications to the problems of robot perception and scene understanding in unstructured environments, and the related problems of localisation, environment mapping and path planning. State-of-the-art sensor fusion methods and multimodal scene understanding approaches are also discussed and evaluated within this context. The paper concludes with an in-depth discussion regarding the current state of the autonomous ground robot navigation challenge in unstructured outdoor environments and the most promising future research directions to overcome these challenges.