Accurate load forecasting in buildings plays an important role for grid operators, demand response aggregators, building energy managers, owners, customers, etc. Probabilistic load forecasting (PLF) becomes essential to understand and manage the building’s energy-saving potential. This research explains a methodology to optimize the results of a PLF using a daily characterization of the load forecast. The load forecast provided by a calibrated white-box model and a real weather forecast was classified and hierarchically selected to perform a kernel density estimation (KDE) using only similar days from the database characterized quantitatively and qualitatively. A real case study is presented to show the methodology using an office building located in Pamplona, Spain. The building monitoring, both inside—thermal sensors—and outside—weather station—is key when implementing this PLF optimization technique. The results showed that thanks to this daily characterization, it is possible to optimize the accuracy of the probabilistic load forecasting, reaching values close to 100% in some cases. In addition, the methodology explained is scalable and can be used in the initial stages of its implementation, improving the values obtained daily as the database increases with the information of each new day.