To sustain competitiveness in contemporary, fast-paced markets, organizations increasingly focus on innovating their business models to enhance current value propositions or to explore novel sources of value creation. However, business model innovation is a complex task, characterized by shifting characteristics in terms of uncertainty, data availability and its impact on decision making. To cope with such challenges, business model evaluation is advocated to make sense of novel business models and to support decision making. Key performance indicators (KPIs) are frequently used in business model evaluation to structure the performance assessment of these models and to evaluate their strategic implications, in turn aiding business model decision making. However, given the shifting characteristics of the innovation process, the application and effectiveness of KPIs depend significantly on how such KPIs are defined. The techniques proposed in the existing literature typically generate or use quantitatively oriented KPIs, which are not well-suited for the early phases of the business model innovation process. Therefore, following a design science research methodology, we have developed a novel method for defining business model KPIs, taking into account the characteristics of the innovation process, offering holistic support toward decision making. Building on theory on linguistic summarization, we use a set of structured templates to define qualitative KPIs that are suitable to support early-phase decision making. In addition, we show how these KPIs can be gradually quantified to support later phases of the innovation process. We have evaluated our method by applying it in two real-life business cases, interviewing 13 industry experts to assess its utility.