The in-house monitoring of elders using intelligent sensors is a very desirable service that has the potential of increasing autonomy and independence while minimizing the risks of living alone. Because of this promise, the efforts of building such systems have been spanning for decades, but there is still a lot of room for improvement. Driven by the recent technology advances in many of the required components, in this article, we present a scalable framework for detailed behavior interpretation. Our framework supports in-house monitoring of elders using an intelligent gateway and a set of cheap commercially available sensors, in addition to more advanced camera-based human localization sensors and a client for GPS-enabled mobile phones that provides monitoring when outdoors. In this article, we report our experiences and present our current progress in three main components: sensors, middleware, and behavior interpretation mechanisms spanning from simple programmable rule-based alerts to algorithms for extracting the temporal routines of individuals.