Towards innovation measurement in the software industry

Abstract

Context: In today's highly competitive business environment, shortened product and technology life-cycles, it is critical for software industry to continuously innovate. To help an organisation to achieve this goal, a better understanding and control of the activities and determinants of innovation is required. This can be achieved through innovation measurement initiative which assesses innovation capability, output and performance.Objective: This study explores definitions of innovation, innovation measurement frameworks, key elements of innovation and metrics that have been proposed in literature and used in industry. The degree of empirical validation and context of studies was also investigated. It also elicited the perception of innovation, its importance, challenges and state of practice of innovation measurement in software industry. Methods:In this study, a systematic literature review, followed by online questionnaire and face-to-face interviews were conducted. The systematic review used seven electronic databases, including Compendex, Inspec, IEEE Xplore, ACM Digital Library, and Business Source premier, Science Direct and Scopus. Studies were subject to preliminary, basic and advanced criteria to judge the relevance of papers. The online questionnaire targeted software industry practitioners with different roles and firm sizes. A total of 94 completed and usable responses from 68 unique firms were collected. Seven face-to-face semi-structured interviews were conducted with four industry practitioners and three academics.Results: Based on the findings of literature review, interviews and questionnaire a comprehensive definition of innovation was identified which may be used in software industry. The metrics for evaluation of determinants, inputs, outputs and performance were aggregated and categorised. A conceptual model of the key measurable elements of innovation was constructed from the findings of the systematic review. The model was further refined after feedback from academia and industry through interviews. Conclusions:The importance of innovation measurement is well recognised in both academia and industry. However, innovation measurement is not a common practice in industry. Some of the major reasons are lack of available metrics and data collection mechanisms to measure innovation. The organisations which do measure innovation use only a few metrics that do not cover the entire spectrum of innovation. This is partly because of the lack of consistent definition of innovation in industry. Moreover, there is a lack of empirical validation of the metrics and determinants of innovation. Although there is some static validations, full scale industry trials are currently missing. For software industry, a unique challenge is development of alternate measures since some of the existing metrics are inapplicable in this context. The conceptual model constructed in this study is one step towards identifying measurable key aspects of innovation to understanding the innovation capability and perfo...

Open innovation using open source tools: a case study at Sony Mobile

“…However, measuring that value using key performance indicators is a daunting challenge. Edison et al (2013) confirmed a limited number of measurement models, and that the existing ones neither model all innovation aspects, nor say what metric can be used to measure a certain aspect. Furthermore, existing literature is scarce in regards to how data should be gathered and used for the metrics proposed in the literature.…”

“…Another benefit is the free maintenance and bug corrections and the test cases extension for further quality assurance. By extension, these software improvements may be labeled as product innovations depending on what definition to be used (Edison et al 2013). This may also be viewed from the process innovation perspective (OECD 2005) as Sony Mobile gets access to extra work force and a broad variety of competencies, which are internally unavailable (Dahlander and Magnusson 2008).…”

Open innovation in software engineering: a systematic mapping study

“…The study conducted by West and Bogers [61] could be considered partly systematic, since the relevance can be seen in terms of data sources, inclusion/exclusion criteria and data extraction. On the other hand, the review conducted by Edison et al [17] adheres to Kitchenham et al [35] and Petersen at al. [45] guidelines.…”

“…We identified four secondary studies (literature reviews) [17,29,61,66] on open innovation, relevant to this study, by applying forward snowball sampling, using the study by West and Bogers [61] as a baseline. The studies are summarized in Table 1.…”

“…However, there is a lack of systematic efforts that focus on summarizing the current state of the literature on the OI relation to SE. Previous reviews are either not systematic [29,66], partly systematic [61] or, for example, focus on the history or evolution of OSS or available innovation metrics [17]. Moreover, these reviews lack quality criteria to support the interpretation of the results in favor or against OI.…”