Fire detection has been an issue of interest to researchers due to its significant damage to lives and property within a very short time. One of the recent solutions developed to detect fire is to use Internetof-Things (IoT) devices equipped with cameras for surveillance. The captured videos of surroundings may be processed by the IoT devices themselves or at the cloud. The latter case is required if the detection algorithm is computationally demanding. However, the use of cloud has a flaw. In fact, using the cloud could pose the threat of having the privacy of a place violated, either through hacking or unauthorized access to the footage of the place where the cloud is installed. In this paper, a fire detection system that preserves the privacy of surroundings, while maintaining a high level of accuracy for fire detection is proposed. The proposed system makes use of the cloud for fire detection; and that is achieved by sending to the cloud features extracted from the video captured by the IoT device, instead of sending the actual footage. Binary video descriptors and Convolutional Neural Network (CNN) have been used to develop the fire detection algorithm. The video descriptors are used to extract features, while the CNN is used for classification. Videos with real fire and non-fire scenes have been used in this development. Results show that the performance of proposed fire detection algorithm can achieve 97.5% classification accuracy, that outperforms the state-of-the art algorithms which make direct use of raw videos. Therefore, the proposed fire detector is as reliable as other available systems, with the advantage of having a privacy-preserving capability. It is also demonstrated that the proposed video descriptors can be implemented for real-time processing using an IoT device, Raspberry Pi 4 platform, with an average processing speed of 100ms per frame, which well satisfies practical needs.