To accomplish more valuable and more accurate video fire detection, this paper points out future directions and discusses first steps which are now being taken to improve the vision-based detection of smoke and flames. First, an overview is given of the state of the art detection methods in the visible and infrared spectral range. Then, a novel multi-sensor smoke and flame detector is proposed which combines the multimodal information of low-cost visual and thermal infrared detection results. Experiments on fire and nonfire multi-sensor sequences indicate that the combined detector yields more accurate results, with fewer false alarms, than either detector alone. Next, a framework for multi-view fire analysis is discussed to overcome the lack in a video-based fire analysis tool and to detect valuable fire characteristics at the early stage of the fire. As prior experimental results show, this combined analysis from different viewpoints provides more valuable fire characteristics. Information about 3D fire location, size and growing rate can be extracted from the video data at practically no time. Finally, directions towards standardized evaluation and video-driven fire forecasting are suggested.