Fall detection became a major concern especially for elderly who lives alone at home. Unexpected situations might happen that influence their health, security and well-being. The development of an intelligent surveillance system is required to alleviate the negative effects of unforeseen circumstances and assisting the elderly in independent living. Currently, convolutional neural network has been successfully used for solving various computer vision tasks, such as object detection and recognition. In this paper, we propose a new vision system for elderly fall detection based on new two stream convolutional neural networks. First, human silhouette is extracted based on background subtraction and person recognition. Second, history of binary motion image HBMI is fed into the first stream characterizing the human shape variations. The second stream is based on amplitude and orientation of optical flow defining the velocity and the direction of the human motion. The system classifies fall events using score fusion schema. Transfer learning is performed to deal with the small amount of fall datasets. Our final network outperforms state-of-the-art results on standard fall datasets.