With the popularity of home service robots (e.g., floor sweepers), robots should be considered to have more elderly care features. Compared to fixed home monitors with a limited field of view, falls detection with service robots is an ideal solution to keep the elderly and disabled people within sight. However, the user's actions such as lying on the bed to sleep or slumping on the sofa to rest, cause the traditional falls detection system to generate false alarms, which disrupts the user's family life. In this article, the improved Faster R-CNN network was proposed by adding temporal action sequences and calculating fall acceleration, which shows the low misjudgment rate on the service robot platform. Firstly, motion images were captured to obtain the target's motion area description and action timing at the input stage. Then, the Faster R-CNN algorithm was implemented to further check the suspected falls based on the falling acceleration of the detected actions during the training phase. Finally, the action mistaken for falling was eliminated by the proposed temporal action sequences module. Network training and robotic platform test results show the proposed method can recognize falls (or false falls) to avoid false alarms. On the service robot platform, experimental results show the FAR is 8.19 and the processing time is 0.79s.