Evaluation and Measurement of Software Process Improvement—A Systematic Literature Review

Abstract

BACKGROUND-Software Process Improvement (SPI) is a systematic approach to increase the efficiency and effectiveness of a software development organization and to enhance software products. OBJECTIVE-This paper aims to identify and characterize evaluation strategies and measurements used to assess the impact of different SPI initiatives. METHOD-The systematic literature review includes 148 papers published between 1991 and 2008. The selected papers were classified according to SPI initiative, applied evaluation strategies and measurement perspectives. Potential confounding factors interfering with the evaluation of the improvement effort were assessed. RESULTS-Seven distinct evaluation strategies were identified, whereas the most common one, "Pre-Post Comparison", was applied in 49% of the inspected papers. Quality was the most measured attribute (62%), followed by Cost (41%) and Schedule (18%). Looking at measurement perspectives, "Project" represents the majority with 66%. CONCLUSION-The evaluation validity of SPI initiatives is challenged by the scarce consideration of potential confounding factors, particularly given that "Pre-Post Comparison" was identified as the most common evaluation strategy, and the inaccurate descriptions of the evaluation context. Measurements to assess the short and mid-term impact of SPI initiatives prevail, whereas long-term measurements in terms of customer satisfaction and return on investment tend to be less used.

Measuring dimensions of Software Engineering projects' success in Academic context

“…Indeed, a systematic literature review of 148 papers published between 1991 and 2008 [27] revealed that Effort and Productivity were defined as success indicators in 63% of studies of process improvement initiatives. In the simplest terms, effort is the time spent by the team during the development process and productivity is its output size in terms of KLOC (kilo lines of code) [28].…”

“…While much research has addressed the definition and measurement of success in industrial software engineering undertakings [3], [4], [5], so far little attention has been paid to the same problem in academic context. This article explores how assessment criteria used for IT deliverables can be translated into an academic grading context and aims to introduce a set of metrics, and measures for evaluation of projects developed by students.…”

“…A thorough literature review reveals three main criteria of project success [3], [5], [6], [7], [9]:…”

A Means-Ends Design of SCRUM+: an agile-disciplined balanced SCRUM enhanced with the ISO/IEC 29110 Standard

“…Such SPMS are core initiatives (i.e. process frameworks besides best practices and tools [7]) into the Software Process Improvement (SPI) approach. According to [7] SPI refers to "a systematic approach to increase the efficiency and effectiveness of a software development organization and to enhance software products".…”

“…process frameworks besides best practices and tools [7]) into the Software Process Improvement (SPI) approach. According to [7] SPI refers to "a systematic approach to increase the efficiency and effectiveness of a software development organization and to enhance software products". However, according to [4] these SPI initiatives "were not written for small projects, small development organizations, or companies with between 1 and 25 employees, and are consequently difficult to apply in such settings".…”