To reduce the risk of elderly people falling in a private room without relying on a closed-circuit television system that results in serious privacy and trust concerns, a fall monitoring system that protects the privacy and does not monitor a person’s activities is needed. An ultrasonic-based sensor system for elderly fall monitoring in a smart room is proposed in this study. An array of ultrasonic sensors, whose ranges are designed to cover the room space, are initially installed on a wall of the room, and the sensors are rotated to transmit and receive ultrasonic signals to measure the distances to a moving object while preventing ultrasonic signal interference. Distance changes measured by ultrasonic sensors are used as time-independent patterns to recognize when an elderly person falls. To evaluate the performance of the proposed system, a sensor system prototype using long short-term memory was constructed, and experiments with 25 participants were performed. An accuracy of approximately 98% was achieved in this experiment using the proposed method, which was a slight improvement over that of the conventional method.