Load shapes obtained from smart meter data are commonly utilized to understand daily energy use patterns for adaptive operations in applications such as Demand Response (DR). However, they do not provide information on the underlying causes of specific energy use patterns -i.e., inference on appliances' time-of-use (ToU) as actionable information. In this paper, we investigated a scalable machine learning framework to infer the appliances' ToU from energy load shapes in a collection of residential buildings. A scalable and generalized inference model obviates the need for model training in each building to facilitate its adoption by relying on training data from a set of previously observed buildings with available appliance-level data. To this end, we demonstrated the feasibility of using load shape segmentation to boost ToU inference in buildings by learning from their nearest matches that share similar energy use patterns. To infer an appliance ToU for a building, classification models are trained for inference on subintervals of load shapes from matched buildings with known ToU. The framework was evaluated using real-world energy data from Pecan Street Dataport. The results for a case study on electric vehicles (EV) and dryers showed promising performance by using 15-min smart meter load shape data with 83% and 71% F-score values, respectively, and without in-situ training.INDEX TERMS Demand response, smart meter, distributed energy resources, segmentation, machine learning, time-of-use, Non intrusive load monitoring (NILM).