The timely detection of terrain drop-offs is critical for safe and efficient off-road mobility, whether with human drivers or with terrain navigation systems that use autonomous machine-vision. In this paper, we propose a joint tracking and detection machine-vision approach for accurate and efficient terrain drop-off detection and localization. We formulate the problem using a hyperstereo camera system and build an elevation map using the range map obtained from a stereo algorithm. A terrain drop-off is then detected with the use of optimal drop-off detection filters applied to the range map. For more robust results, a method based on multi-frame fusion of terrain drop-off evidence is proposed. Also presented is a fast, direct method that does not employ stereo disparity mapping. We compared our algorithm's detection of terrain drop-offs with time-code data from human observers viewing the same video clips in stereoscopic 3D. The algorithm detected terrain drop-offs an average of 9 seconds sooner, or 12m farther, than the human observers. This suggests that passive image-based hyperstereo machine-vision may be useful as an early warning system for off-road mobility.